To be continued

In case you're worrying that the series about Erlang/OTP won't be continued be assured, that I'll add more content sure. But beside several other tasks I'm currently making some progress in migrating my root server as well as other web locations on two new servers - cheaper but also better. And here I'll also consolidate the web presentation, the old wiki, and this blog. I'll notify you soon.

Software Architecture with Erlang/OTP - Part 7 - Supervision Trees

In some parts Erlang is like the real life: there are many processes which do the real work and some which are supervising them. And there's a hierarchy. As I said, real life. *smile*

Typically Erlang/OTP applications are seperated into worker processes, which are doing their jobs, and supervisor processes, which monitor the worker processes. They get notified in case of a terminating worker. Beside the process id the reason for the termination is part of this signal. This way it is easy to implement a fault tolerant behaviour where a supervisor is able to restart an abnomally terminating worker process he's responsible for. Because supervisors are just processes too they can also be supervised. That's needed to build modular systems with a clear separation of concerns.

Supervision trees can be build manually through linking processes with the function link(Pid) or when spawning it with spawn_link(Fun). Normal terminating processes will cause no special reaction, but abnormal terminating processes lead to the termination of all linked processes. This may already help in many situations, but there are more possibilities. Calling process_flag(trap_exit, true) inside the process that should act as a supervisor leads to transformation of signals into regular messages on the format {'EXIT', Pid, Reason}. Now it's possible to receive these messages and react, e.g. through restarting the terminated process.

That's a standard situation. So it's no wonder that the OTP contains a module for this job. And the module name supervisor doesn't wonder too. *smile* Like the generic server and the generic finite state machine the supervisor needs a callback module. This time it only has to provide one function called init(Args). The result is a restart strategy, values to define the restart frequeny, and a list of child specifications. The list of the supervised childs can also be changed through adding and deleting child specifications at runtime. Each child specification is a tuple containing an identificator, the function to start with module, function name, and arguments, a restart strategy, a shutdown strategy, a flag if the process is a worker or itself a supervisor, and a list of depending modules for the release handler. This list already shows that there is much room to configure a supervisor.

So for example the Tideland EAS supervisor is starting and stopping channels with

%% @spec (atom(), atom(), pid()) -> {ok, Pid} | {error, Reason}
%% @doc Start of a channel.

start_channel(EasModule, ChannelId) ->
    StartFunc = {EasModule, start_supervised, [ChannelId]},
    ChildSpec = {{channel, ChannelId}, StartFunc, transient, 10000, worker, [EasModule]},
    supervisor:start_child(?MODULE, ChildSpec).

%% @spec (atom()) -> ok | {error, Reason}
%% @doc Stop of a channel.

stop_channel(ChannelId) ->
    supervisor:terminate_child(?MODULE, {channel, ChannelId}),
    supervisor:delete_child(?MODULE, {channel, ChannelId}).

The exact configuration is very well documented. But there's one last question. How to maintain the state of a restarting process? You already know the comfortable way the gen_server provides. But the state gets lost if the server process restarts. So in this case it's a common pattern to use a special second process, an ets table, or a Mnesia ram only table. But this overhead is only needed in case of process states that can't start at zero. If server processes work stateless and e.g. use a database as backend it can be kept simple.

Software Architecture with Erlang/OTP - Part 6 - gen_fsm

Beside the generic server gen_server one of the most practical modules for own processes is the finite state machine gen_fsm. It is very similar to gen_server regarding the implementation of the user functionallity inside a callback module and its options for tracing, logging, and statistics, timeouts and hibernation, clean terminations and the handling of code changes. This makes it once again comfortable to concentrate on implementing the business logic without hassling with many runtime aspects.

In addtion to the standard behaviour callback functions of the finite state machine, take a look at the documentation, the developer has to implement additional functions for each state. Think about a process building a bridge between the two external systems foo and bar. First both shall be connected using connector processes specified through special modules, then the connector processes poll them and notify the bridge for transfering the data after a conversion. If one of both signals a stopping the bridge shall end its work.

init({FooModule, BarModule}) ->
    {ok, FooPid} = FooModule:start(FooCfg, self()),
    {ok, BarPid} = BarModule:start(BarCfg, self()),
    {ok, awaiting_connection, {FooModule, false, BarModule, false}}.

awaiting_connection({connected, foo}, {FooModule, false, BarModule, false}) ->
    {ok, awaiting_connection, {FooModule, true, BarModule, false}};
awaiting_connection({connected, foo}, {FooModule, false, BarModule, true}) ->
    FooModule:poll(),
    BarModule:poll(),
    {ok, polling, {FooModule, BarModule}};
awaiting_connection({connected, bar}, {FooModule, false, BarModule, false}) ->
    {ok, awaiting_connection, {FooModule, false, BarModule, true}};
awaiting_connection({connected, bar}, {FooModule, true, BarModule, false}) ->
    FooModule:poll(),
    BarModule:poll(),
    {ok, polling, {FooModule, BarModule}}.

polling({received, foo, Data}, {FooModule, BarModule}) ->
    BarModule:send(foo2bar(Data)),
    {ok, polling, {FooModule, BarModule}};
polling({received, bar, Data}, {FooModule, BarModule}) ->
    FooModule:send(bar2foo(Data)),
    {ok, polling, {FooModule, BarModule}};
polling(stop, {FooModule, BarModule}) ->
    FooModule:stop(),
    BarModule:stop(),
    {stop, normal, {FooModule, BarModule}}.

This is a trivial example, but it already shows one of the great strength of gen_fsm. It is really simple in Erlang to use concurrent running processes for different tasks, e.g. using the generic server. But if you do only synchronous calls there's no win opposite to sequential programming. When using asynchronous cast messages instead it's difficult to handle the also asynchonously resent results. And here the finite state machine helps a lot. A business process is a kind of state machine too. So it can be implemented using the gen_fsm module. Parallel activities can be started once and the results collected before continuing with the next step.

As already mentioned above the finite state machine knows event timeouts like the generic server. But sometimes a business process has an overall maximum runtime. So it is possible to start a timer with gen_fsm:start_timer(Time, Msg) at any time and tell it to send the defined message inside a timeout event when the time is over. Depending on the current state this event can be handled inside the callback module.

The combination of gen_server and gen_fsm is like the combination of services and business processes. Both work very good together. How they are supervised at runtime I'll describe in the next entry. So, stay tuned!

Software Architecture with Erlang/OTP - Part 5 - gen_server

The next entries I'll cover the most important modules of the OTP, today the generic server behaviour gen_server. It is the base module for server processes with a standard set of functions and functionallity. This allows to concentrate on implementing the business functions while inherit many useful features like tracing, logging, and statistics, synchronous asynchronous communication, management of process states, and handling of code changes.

Take for example a little counter process. It is implemented as a callback module:

-module(counter).
-behaviour(gen_server).
 
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).
 
init({InitalCounter, Step}) ->
    {ok, {InitialCounter, InitialCounter, Step}}.
 
handle_call(get, From, {Counter, InitialCounter, Step}) ->
    {reply, Counter, {Counter, InitialCounter, Step}}.
 
handle_cast(count, {Counter, InitialCounter, Step}) ->
    {noreply, {Counter + Step, InitialCounter, Step}};
handle_cast(reset, {Counter, InitialCounter, Step}) ->
    {noreply, Counter, {InitialCounter, InitialCounter, Step}};
handle_cast(stop, {Counter, InitialCounter, Step}) ->
    {stop, normal, {Counter, InitialCounter, Step}).
 
 handle_info(Info, State) ->
    {noreply, State}.
 
terminate(Reason, State) ->
    ok.
 
code_change(OldVsn, State, Extra) ->
    {ok, State}.

It will be instantiated and used with

{ok, Pid} = gen_server:start(counter, {0, 5}, []),
 
gen_server:cast(Pid, count),
gen_server:cast(Pid, reset),
gen_server:cast(Pid, count),
gen_server:cast(Pid, count),
 
Result = gen_server:call(Pid, get), % Returns 10.
 
gen_server:cast(Pid, stop).

Instead of calling the gen_server functions directly they are typically wrapped. This hides the mechanism and makes it more user/developer friendly and maintainable. So far, so good, but nothing special. But it gets more interesting with all the additional features of the gen_server. So it is possible to add a local or a global name of the process so that processes can be addressed more easy. All using processes can directly send messages to the registered process. And it's possible to send message to locally registered processes on multiple node parallely. Other start arguments control how long the initialization may last before it is cancelled or if the process shall be observed through the sys module. So events can be logged or trigger user defined higher order functions. Additionally system events can be traced and statistical informations retrieved.

Until now I've only described the communication on demand. But sometimes it may be interesting that a process reacts on idle time. E.g. when the process has to work for a few minutes each hour but the other time the usage of resouces shall be reduced. Here the gen_server provides to interesting features. So if all messge answers also contain a timeout value it will be taken to send a special message to itself after the according idle time.

handle_call(Msg, From, State) ->
    % Also return a timeout value in milliseconds.
    {reply, Value, NewState, Timeout}.

handle_info(timeout, State) ->
    % React on the timeout, e.g. hibernate the process.
    {noreply, NewState, hibernate}.

Additionally the timeout can be used otherwise, e.g. for background calculations, checking caches, or lazy writings. And also the hibernation can be done on demand through return hibernate as part of a handle_call / handle_cast return.

It's easy to see how flexible the gen_server is. But it also has some drawbacks. When doing synchronous calls there may be a dependency between process like A calls B, B calls C, and C calls A. Then, without a specified timeout, a deadlock happens. With a timeout this can be prevented, but the behaviour is not the wanted one. I'll show later in this series how this situation can be handled. Another problem may be registered processes. Here only one instance exists per node or global. So it may be a bottleneck when heavily used. In this case only a kind of broker should be registered and dispatch the messages to special worker processes. But in this case you've got realize a special way for the process state. And this solution should not be a bottleneck, too.

You'll find more informations in the Erlang documentation, tomorrow I'll cover gen_fsm. So, stay tuned!

Software Architecture with Erlang/OTP - Part 4 - The Language

First of all, don't expect an Erlang tutorial here. If this is needed I recommend the book "Programming Erlang: Software for a concurrent word" by Joe Armstrong, the inventor of Erlang. But I want to highlight some special features of Erlang that have a great influence on the design of many modules and libraries, and so naturally also on the architecture of the Tideland EAS.

Erlang is a functional language, not as pure as e.g. Scheme, but still functional. The arity of a function may differ, also the types of the arguments due to the dynamic typing. Pattern matching and guards, which I'll describe below, help to handle this flexibility in a very powerful manner. Additionally Erlang functions allow tail recursive calls. This is very important because the language doesn't know loops. Instead recursive functions are used, even for endless loops which are used in processes. In this case there will be no stack overflow, even if those still can be produced. *smile* The organization of functions is done using modules, public functions have to be exported explicitely. From other modules they are called using my_module:add(1, 2). The nice fact here is that both parts of this call, the module and the function name, can be replaced with variables. This is a very useful feature for the implementation of callback modules. They allow to change the behaviour of functions. For example take the processing of some data where you want to be flexible in the how it is persisted.

persistent_process(Id, PersistencyHandler) ->
    OriginalData = PersistencyHandler:load(Id),
    ProcessedData = process(OriginalData),
    PersistencyHandler:store(Id, ProcessedData).

Now different modules can be implemented with exported functions load/1 and store/2, e.g. one using the dets module, one using the Mnesia database, or any own implementation. They have to be passed in the function call:

persistent_process(4711, dets_persistency).

Modules like supervisor and gen_server use this mechanism called behaviour. The finite state machine gen_fsm even manages also the current state inside a variable and calls the according callback function:

dispatch({'$gen_event', Event}, Mod, StateName, StateData) ->
    Mod:StateName(Event, StateData);

This way the implementation of functional and non-functional features can easily be separated, generic modules build the fundament for reusage.

Two of the features I've already mentioned above and that I don't want to miss are pattern matching and guards. They are used to choose one of a set of different implementations of the same function depending on the passed argument. They work the same way when choosing the right actions inside of receive, case, and if statements.

calc(discount, Amount, AnnualTurnover) when AnnualTurnover > 5000 ->
    {ok, Amount * 0.9};
calc(discount, Amount, AnnualTurnover) when AnnualTurnover > 1000 ->
    {ok, Amount * 0.95};
calc(discount, Amount, AnnualTurnover)  ->
    {ok, Amount};
calc(comission, Amount, AnnualTurnover) when Amount > 500 ->
    {ok, Amount * 0.05};
calc(comission, Amount, AnnualTurnover) ->
    {ok, Amount * 0.02};
calc(Any, _Amount, _AnnualTurnover) ->
    {error, {unknown_operation, Any}}.

So when called as calc(discount, 100.0, 2750.0) the result is the tuple {ok, 95.0}, and the call of calc(comission, 100.0, 2750.0) leads to {ok, 2.0}. Any other operation than discount or comission leads to an error. While there are many situations where this feature is helpful it especially helps when developing a library with behaviours:

dispatch(Action, From, Handler, State) ->
    case Handler:process(Action, State) of
        {reply, Data, NewState} ->
            From ! {ok, Data},
            {ok, NewState};
        {ok, NewState} ->
            {ok, NewState};
        {error, Reason} ->
            From ! {error, Reason},
            stop
    end.

When developing a handler module the function process/2 has to be implemented with different patterns for the action:

process({add, Principal, Description}, State) -> ...;
process({set_credential, Principal, Credential}, State) -> ...;
process({deactivate, Principal}, State) -> ...;
process({activate, Principal}, State) -> ...;
process({authenticate, Principal, Credential}, State) -> ...;

Here we see another nice feature, tuples. It's very often needed to pass or return more than one value. But it is inconvenient to define special records or classes each time. In Erlang it is very simple to create tuples and to retrieve values out of them using pattern matching. Tuples and lists together allow real powerful data structures.

So, what's left? Processes, the most important feature of Erlang. The are lightweighted in starting, stopping, and at runtime. And they are isolated, what's extreme important for the reliability of Erlang applications. A process can just to a one time job and terminates when it is finished. Or it can be a longer running process responsible for different tasks, see the authentication example. But how does it know what to do when it is isolated? Each process has an own message queue and when starting a function as a new process a process id will be returned. This id, or a registrated alias name, can be used as the target for a message sending. Messages can be received using the receive statement and pattern matching. So a process loop function may be:

loop(Handler, State, Timeout) ->
    receive
        {From, Action} ->
            case dispatch(Action, From, Handler, State) of
                {ok, NewState} -> loop(Handler, NewState, Timeout);
                stop -> stop
            end;
        stop ->
            stop
    after
        Timeout ->
            case Handler:timeout(State) of
                {ok, NewState} -> loop(Handler, NewState, Timeout);
                stop -> stop
            end
    end.

The recursive called loop statements inside the function let it run endless until it is stopped explicitely or as the result out of the dispatch function. Now a message can be sent through

Pid ! {self(), {add, MyPrincipal, MyDescription}}.

The self statement passes the process id of the calling process. This is needed to resend the result. A process never handles two messages at the same time, they are processed one after the other. Statements like that are very often encapsulated in public functions of the process module:

Result = authentication:add(Pid, MyPrincipal, MyDescription).

The function also contains the receive statement to get the result from the process. Annother feature you can see above is the timeout inside the receive. When there's no message received inside the defined timespan in milliseconds the following statements will be executed. That's helpful for automation and needed for a non-blocking process flow.

I hope these first impressions of Erlang do help. The next blog entries will be shorter and cover the major OTP modules which are based on the above mentioned features. Enjoy it.

Software Architecture with Erlang/OTP - Part 3 - EAS Overview

The architecture of the Tideland EAS is mainly organized around four major modules:

• The bus is the one face to the user (developer, administrator) of the EAS. There's no real bus, but one process per node managing the supervisor, the registry, and the configuration. Additionally this module provides functions for starting and stopping of bus, channel, and service, for subscribing and unsubscribing, for the start of business processes, and for the raising of events.


• The channel is used by the bus. It just manages its own service subscriptions and passes the received events to those services. There's one instance running per channel. It does no filtering of the content, this has to be done by the services themselves, e.g. using pattern matching and guards.


• The service manages it's callback module and the service status. The latter is set only once during the initialization of the service. This is because the real work is done inside a process ring buffer provided by the Tideland CEL. So multiple requests can be handled concurrently. If real states are needed the service itself has to care, e.g. through the usage of the ets module. There's one instance of the service process running per service.


• And at last there's the business process manager. This one is different from the three modules above because an instance will be created for each new process and it is terminating when the process ends. Internally it is implemented as a state machine and also the callback modules which are implementing the real business process are working as state machines.

Beside those modules there are some additional helpers. But the bus, the channel, the service, and the business process manager are the most important ones and the way how they use and communicate to each other is the core of the EAS.

Before describing how they are implemented and why I've choosen this way I would like to give a very little introduction into the language Erlang - no real tutorial, just the most important features that are also responsible for the EAS architecture - and into the major modules of the OTP. This will be the content of the next blog entries, so stay tuned!

Software Architecture with Erlang/OTP - Part 2 - Motivation

When starting a private project it's helpful to have a vision in mind. In my case there are two projects I'm carrying with me for a longer time: the first one is a mixture of a semantic wiki and a blog, the second one a special community portal I'll surely write about later. Meanwhile there's a vague third idea, but it's too early to tell more about it. *smile* Beside those projects I want to learn more about Erlang, about how to do concurrency-oriented programming - I'm doing OOP now for more than 20 years - and how to realize an event-driven architecture.

So I've started with my typical bottom-up approach, first developing a library with useful stuff (the Tideland Common Erlang Library), then the server (Tideland Events and Services), and as the last preparing step, some always useful services (Tideland Common EAS Services). Those three projects build the foundation for the real projects mentioned above.

The major part surely is the EAS. The major design goals for this software are:

1. Following the event-driven paradigm of raising events in different sources (front ends, business processes, adapter to external systems, and more) and processing them in subscribed services.


2. Multiple subscribers should be able to handle the same event.


3. Supporting a loose coupling between and a high cohesion services.


4. Flexible assignment of callback modules to services and subscription of services to channels through configuration.


5. Using typical erlang mechanisms like behaviours for an easy implementation of services and business processes.


6. Technical aspects like distribution and pooling are transparent for the developer of services abd business processes.


7. Simple API, pure Erlang, and no XML for configuration. *smile*

The EAS tries not to be the next great application server, and it doesn't want to be an even greater integration platform. But I want it to be a solid and flexible foundation for scaleable Erlang applications on the server.

So next time I'll write about the general software architecture and the modules.

Software Architecture with Erlang/OTP - Part 1 - Preface

During the last months I've developed the base for my future Erlang applications, the Tideland Events and Services (EAS) server. It's not yet finished, but I've learned a lot about how to use the concurrency in Erlang, pitfalls, tricks for stability and maintainability, and several of the standard Erlang modules. There are still some blank spots, but I'm exploring them more and more.

So I decided to publish a little series of articles about software architecture with Erlang/OTP here in my blog. It's not a real tutorial, but I would like to share the experiences I made and why they led to some of my design decisions. Starting point is my motivation for the development of the EAS, then an overview of the major OTP modules and how I'm using them, where to use concurrency, ways to handle the control flow in an concurrent environment, robustness and exception handling, concepts for flexibility and simplicity, working with functional domains, persistency with Mnesia, and the deployment of Erlang applications. What's still open is the topic web frameworks. Here I'm still not really sure how I'll do it. There are two or three ideas in my head, all with pros and cons. We'll see.

Currently I don't know how granular this series will be, how deep, and how log. This surely will also depend on your feedback, I appreciate it. So, stay tuned!

2009 whisky tasting season started

Yesterday the 2009 whisky tasting season started at Kottkamps. I've really looked foreward to it and it has been nice to see our community again. This year there will be some more events due to the part-time shop Sandra and Ulli opened. So the next tasting together with Scottish cheese, salmon, and chocolate will be on Feb, 27th.

This time I've first choosen a Benriach 12y Jamaica Rum due to the Rum finish of one of my favorite malts, the Glenfiddich 21y Havanna Reserve. First it hasn't been so good, with raisins, but too muddy and mouldy. It became better over the time, but not really good. For the second one I've choosen an Edradour 1996 11y Rum, once again a chance for a malt with a Rum finish. It has definitely been better, for an Edradour very mild in nose and flavour, warm and smooth, and with no peat and smoke in the long lasting finish.

It has been a fine selection, but topped through number three, a Caol Ila 1995 Distillers Edition. This malt will also be part of my collection in future. A sweet and peaty nose, tastes of spices, malt, and smoke, and a long, warm, complex finish. That's how I like it. To close the evening I've choosen another Islay, the Laphroaig 2000 7y. Nedless to say anything, a classical and wild Islay, powerful. A wonderful finishing.

On OOP, COP and the Tideland EAS

My current software project is the Tideland EAS (Events and Services) as an Erlang/OTP based platfrom for my future server-side applications. To understand the design motivation behind the EAS it may be useful to get a little insight into my software development experiences.

I'm now doing software development since about 25 years, the last 20 years the object-oriented way. So my development style is mostly characterized by my personal understanding of OOP. The bundling of attributes and behaviour is a natural way of thinking and easy to understand unless the imperative way of programming isn't etched on the memory too hard. Depending on the experience of the developer it's more or less simple to determine the right candidates for pure helper classes, for those building the foundation for the own system, and those representing the business model of an application. A large number of design patterns helps to find solutions based on a common understanding.

One of the most important parts of object-orientation is the idea of message passing between instances. When developing Smalltalk Alan Kay has been heavily inspired by Carl Hewitt's research on the actor system for massively parallel, distributed, computer systems. Unfortunately instances in languages like Smalltalk, Java, or C# aren't massively parallel. So messages sent to an instance block the caller until the work is finished. *sigh*

Think about a restaurant, where the guests have to wait for their waiter who is then waiting for the cook. The cook is then cooking the meal, one part after the other, hands it out to the waiter who is then handing it out to the guests. During this the waiter can't attend on other guests. So the restaurant would need a pool of waiters to handle every guest parallely. But additionally it would also need enough cooks to serve all orders.

To allow an environment where at least such a concurrency with pools is possible the traditional environments use threads. While those are relative lightweight from the operating systems point of view they are still very heavy on the instuction level. Sadly threads are not native for objects, they are orthogonal, a kind of meta level. So instances can talk to other instances in different threads without any problem while they are working on another task. Semaphores and synchronized blocks shall help here, but the more complex a system gets and the more a development team size grows the easier it is to confuse the system, to let it get more unstable. No good basis for reliable systems. Here I'm thinking about our waiter from the example above. Guests and cooks are calling for him while he is trying to satisfy all in parallel. It's easy to imagine how fast this poor guy gets confused.

Here I come to the concurrency oriented programming, actor systems, and Erlang. While in traditional OOP languages the default behaviour is synchronous and asynchronicity has to be added manually or through special frameworks it's different in Erlang. Function calls are also snchronous, but the exchange of messages between concurrent running processes is asynchronous. Blocking calls can be simulated through an immediate waiting for the reply sent back from the process. The OTP provides functions for this inside the standard modules. Looked at that way COP seems to be very close to the original idea of OOP. In case of Erlang this gets combined with its functional paradigm, transparent distributed environments, immutable variables, no shared data, pattern matching, guards, and the integrated database Mnesia. And even if the syntax seems to be weird for the curly braced fraction it's due to the Prolog roots more clean than the actual Java or C# standards - even though Smalltalk and Scheme are still more elegant and compelling. *smile*

Those very positive characteristics made me chosing Erlang as the base for my own development. But I want a bit more than the standard OTP provides, the reason for me to develope the EAS. The name Events and Services already unveils the idea of services implementing the functionality of business domains and communication between them in the form of events. So internally the EAS is a publish/subscribe architecture. The services subscribe to topics on channels and front ends, adapters, and services publish their business events together with a topic on a channel. Some events will be handled totally by one subscriber, others will be a flow of events, serial and parallel, between multiple services. Even typical calls will be just a flow of events. The initial event may be new_customer together with the customer data, the answering event could be customer_added together with the processed data. The publisher of this event may also be a different service than the initial receiver of the first event.

The first benefit of this approach is an extreme loose coupling - good for maintenance, very good for extensibility, and even better for scalability. But there's also a second aspect that's driving my design. Typically only one class or component is receiving a request, publish/subscribe architectures allow multiple receivers, so does the EAS. First this is interesting for automatic concurrent activities. E.g. each time a contract gets cancelled the system parallely updates the customer file, tells the financial accounting to strike a balance, and notifis the customer about the receiving of the cancellation. Or inside an order process the event order_ready_for_fulfilment could start the shipping and the generating of the invoice which do their work independently. But by far more interesting is the Complex Event Processing (CEP) with different types of analysis. Events may be checked that they always occur in pairs, with a specified rate inside a time span or a time period, their order is in a specified sequence, they happen in combination, or they are matching specific conditional expressions. Based on rules those conditions may be associated and correlated in domains, their evaluation leads to the generation of new events for an automatic reaction, alerts may be rised early due to detected signs of risks, it leads to a non-technical monitoring of the environment, the understanding of the dynamic nature of the business may be improved.

I've used the subjunctive here a lot, consciously, because the benefit isn't for free. First detecting the right business domains for the implementation as services and to not build a too fine granularity isn't easy. And second detecting the right correlation domains together with the optimal ruleset is a hard job and one has to think hard and long if it fits for the own situation. I'll see which experiences I'll make when the systems are up and running. But it definitely it's interesting and worth a try.