tag:blogger.com,1999:blog-83363183659406196692024-03-14T05:35:11.169+01:00Horsdal ConsultChristian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.comBlogger139125tag:blogger.com,1999:blog-8336318365940619669.post-20016519612841016032022-12-16T08:42:00.007+01:002023-01-12T22:10:09.454+01:00Effective Testing in Microservice Systems talk from NDC Oslo<p> My talk from this years NDC Oslo isnow available on youtube, so in case you missed it: Enjoy.</p><p><br /></p>
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube-nocookie.com/embed/8XbuJ_DczL8" title="YouTube video player" width="560"></iframe> <div><br /></div><div> And you can flip through the slides here:</div><div><br /></div><div> <iframe frameborder="0" height="400" marginheight="0" marginwidth="0" scrolling="no" src="https://www.slideshare.net/slideshow/embed_code/key/rRJYySdc5nD0GK?hostedIn=slideshare&page=upload" width="476"></iframe></div>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-26193707808353229522021-11-05T16:58:00.008+01:002021-11-05T17:11:51.362+01:00Recording of stream about collaboration between microservices<!--Load the Twitch embed script-->
<script src="https://player.twitch.tv/js/embed/v1.js"></script>
<p> The other I was on the Manning Twitch talking about collaboration between microservices. You can see recording of stream below. I was there on occasion of the publication of the second edition of my Microservices book, which only just landed in hard copy a few days before. Much of the material in this talk appears in further detail in the book, though I did change it up a bit for the talk.</p><p>Hope you enjoy it!</p><p><br /></p>
<!--Add a placeholder for the Twitch embed-->
<div id="twitch-embed"></div>
<!--Create a Twitch.Player object. This will render within the placeholder div-->
<script type="text/javascript">
new Twitch.Player("twitch-embed", {
video: "1195598440",
width: "100%",
height: "400",
autoplay: false
});
</script>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-27000977345252828202021-02-10T14:57:00.002+01:002021-02-10T14:59:49.742+01:00Interview on Adventures in .NET<p> I recently had the pleasure of chatting with the folks on the Adventures in .NET podcast and now the episode is out. We talk about some of the things I consider essential to get right to have success with microservices - like getting the responsibilities right, having good observability, good automation and more. We also talked about when to use microservices and when not to. After all, one does not simply microservice. Anyway, the episode is out and you can find it <a href="https://devchat.tv/adventures-in-dotnet/net-055-microservices-or-should-they-be-called-single-responsibility-services-w-christian-horsdal/">here</a>.<br /></p>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-23924535205182521122020-08-04T07:11:00.010+02:002020-08-04T07:14:11.243+02:00Interview about Microservices on the Cynical Developer PordcastI recently had a chat with James Studdart, host of the Cynical Developer podcast about microservices and the <a href="https://www.manning.com/books/microservices-in-net-core-second-edition" name="https://www.manning.com/books/microservices-in-net-core-second-edition">second edition</a> of my microservices book. <a href="https://cynicaldeveloper.com/podcast/146/" name="https://cynicaldeveloper.com/podcast/146/">The podcast is now out</a> and you can listen to it here or any of the usual places you find podcasts:<br /><br />
<audio controls="controls" id="podcastPlayer" src="https://traffic.libsyn.com/cynicaldeveloper/episode-146.mp3?dest-id=461505" style="width: 100%;" type="audio/mp3"></audio>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-81062050070095108172020-01-08T14:33:00.002+01:002020-01-08T14:33:36.463+01:00"Lessons learned working remotely across timezones" video from BuildStuff 2019At BuildStuff 2019 I got the chance to do a talk about the things I've learned from doing remote work for a client. It's a different kind of talk for me. It's not technical, it's about the hard people oriented stuff.<br />
<br />
<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube-nocookie.com/embed/TjPsw0ROF4I" width="560"></iframe>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-19915865971382806132019-10-11T08:36:00.000+02:002019-10-11T11:43:15.608+02:00Identifying and Scoping Microservices talk from DevConfPLI recently had the pleasure for delivering my Identifying and Scoping Microservices talk at DevConf in Krakow. Here's the video. Enjoy 😉<br />
<br />
<br />
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/ogV-X08TR68" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-46827968189111155032019-09-20T08:30:00.002+02:002019-09-20T08:30:29.362+02:00A pattern for synchronizing data from a legacy system to microservices<h2>
Abstract </h2>
A recurring need in the teams I work with seems to be to move and continually synchronize data from a legacy system - typically a monolith - to a new microservice based replacement. In this post I will outline a data synchronization solution that allows<br />
<ul>
<li>The microservice side to catch up on historical data</li>
<li>Synchronizes new data to the microservices as it is produced in the legacy system</li>
<li>Lets us control the speed of synchronization and as a result the load on the legacy system</li>
<li>Is repeatable, so the same data can easily be synchronized again if need be</li>
</ul>
In the interest of keeping this post short I will only show a high level view of the solution.<br />
<br />
<ul>
</ul>
<h2>
Microservices need legacy data</h2>
When replacing an existing legacy system with a modern microservice based system teams (sensibly) tend to do so in steps following the <a href="https://martinfowler.com/bliki/StranglerFigApplication.html">strangler pattern</a>. This means that for a while both systems run in production. Some functionality is handled in the legacy system and some in the microservices. At first very little is handled in the microservices but gradually more and more functionality is handled there. To support those small early steps implemented in the microservices data from the legacy side is often needed. For instance, in a back office system a team in the process of moving to a microservice architecture they might want to implement a new sales dashboard with microservices. To do so we will need order data and possibly other data too, but let's just focus on the order data for now. Orders are still being taken in on the legacy side, so order data is being produced on the legacy side. But the new sales dashboard needs both historical orders and new orders to work correctly. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLghB9Ry_T76RiX8Vg6oUP5I2KrnyxtN5xhucxJKK8ci3p1mHcgVgIVOpq1WJwOr1r6HkQjNBBaEygR64OyMObatSW_0OBTa7FicvYr6Jtg-XLDpeIIsX01Lz7r5HCXq8jrh0I1Fcc0PM/s1600/data+syncs+-+situation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="708" data-original-width="1476" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLghB9Ry_T76RiX8Vg6oUP5I2KrnyxtN5xhucxJKK8ci3p1mHcgVgIVOpq1WJwOr1r6HkQjNBBaEygR64OyMObatSW_0OBTa7FicvYr6Jtg-XLDpeIIsX01Lz7r5HCXq8jrh0I1Fcc0PM/s640/data+syncs+-+situation.png" width="640" /></a></div>
<br />
To make things more interesting let's say the legacy system and the microservices are in different data center - maybe the system is moving from on prem to the cloud as part of the microservice effort.<br />
<br />
<h2>
Solution: A data pump </h2>
A solution to the situation above is to implement a data pump that sends any updates to relevant data in the legacy database over to the microservices. In the example that means new orders as well as changes to orders.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQVfDc0hjB8ona-FOq61P1bZ-pN7yGCAOEuuH9_XhpFi_6wImmbGbxrZOIW6vHBG2AJAXvHd6qXiNROyd9Wz9jsBpTAwID8haqt3yx7IivTQ4vpe9hR0hK5IYpQBhaqufPRrzgjcdNRmQ/s1600/data+syncs+-+solution.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="666" data-original-width="1600" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQVfDc0hjB8ona-FOq61P1bZ-pN7yGCAOEuuH9_XhpFi_6wImmbGbxrZOIW6vHBG2AJAXvHd6qXiNROyd9Wz9jsBpTAwID8haqt3yx7IivTQ4vpe9hR0hK5IYpQBhaqufPRrzgjcdNRmQ/s640/data+syncs+-+solution.png" width="640" /></a></div>
<br />
This solution has two components: A data pump which is deployed in the legacy environment and a data sink which is deployed in the microservices environment. The data pump tracks which data has already been sent over to the microservices and sends new data over as it is produced in the legacy system. The data sink simply receives the data from the pump and posts it onto a queue. This enables any and all microservices interested in the data - e.g. new or updated orders - to subscribe to such messages on the queue and to build up their models of that data in their own databases.<br />
<br />
With the described solution in place any historical data can be sent over to the microservices. That may take a a while, especially if the legacy database cannot take to much additional load. In such cases the data pump can be throttled. Once the pump has caught up sending over historical data it will continue to send over new data in near real time.<br />
<br />
If we put a bit of up front design into the data pump we can also support restarting the process - the pump tracks what it has already sent, resetting that tracking will make it start over. That's sometimes useful if we e.g. don't get the receiving mciroservice right in the first attempt, or if we introduce new microservices that also need data.<br />
<br />
This is a solution I have seen used with success in several of my client's systems and that I think is applicable in many more systems too.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-47408295707830797932019-06-17T18:35:00.000+02:002019-06-17T18:35:34.530+02:00InMemoryLogger available on NuGetI recently made a small library, <a href="https://github.com/horsdal/InMemoryLogger">InMemoryLogger</a>, to support recording and inspecting logs made from .NET Core code under test. The library is pretty small and simply implements a .NET Core <span style="font-family: "Courier New", Courier, monospace;"><a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.ilogger?view=aspnetcore-2.2">Ilogger </a></span>that records incoming logs and exposes them for inspection through a few public properties. I've been using InMemoryLogger for a while now and I think it is ready for other people to use too.<br />
<br />
If you follow arrange/act/assert in your tests the idea is to create the in memory logger in the arrange part either by directly newing it or adding it to the application under tests <span style="font-family: "Courier New", Courier, monospace;">IServiceCollection</span>. Then do whatever to the application under test in the act part, and finally inspect the properties that expose recorded logs on the InMemoryLogger in the assert part.<br />
For more info check out the <a href="https://github.com/horsdal/InMemoryLogger/blob/master/README.md">readme</a> or grab <a href="https://www.nuget.org/packages/InMemoryLogger">InMemoryLogger from NuGet</a>.<br />
<br /> Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-9632677836255698582019-04-03T11:49:00.001+02:002021-11-15T08:21:52.337+01:00Interview on the .NET Core ShowLast week the <a href="https://dotnetcore.show/">.NET Core Show</a> published an <a href="https://dotnetcore.show/episode-23-microservices-in-net-core-with-christian-horsdal/">interview with me</a> where where I talk about various aspects of developing microservices with .NET Core, though most of it technology stack agnostic. Enjoy.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com2tag:blogger.com,1999:blog-8336318365940619669.post-20403999763178959512019-02-16T08:35:00.000+01:002019-02-16T08:38:33.886+01:00Building a self-contained .NET Core app in a containerLast week I needed to<br />
<ol>
<li>Build a <a href="https://docs.microsoft.com/en-us/dotnet/core/deploying/#self-contained-deployments-scd">self-contained .NET Core app</a>.</li>
<li>Not install the .NET Core SDK on the CI server</li>
<li>Fit this into a build pipeline that was already <a href="https://docs.microsoft.com/en-us/dotnet/core/docker/docker-basics-dotnet-core#dockerize-the-net-core-application">containerized</a>.</li>
</ol>
These three requirements led me to build the self-contained app in a container. Doing that is a 2 step process:<br />
<ol>
<li>First build a container using docker build</li>
<li>The run the container with a local folder called "output" as a <a href="https://docs.docker.com/storage/volumes/">volume</a></li>
</ol>
The result is that the self-contained app is in the local "output" folder.<br />
This is a Dockerfile that allows this:<br />
<br />
<script src="https://gist.github.com/horsdal/8a42ca27586a2bb51af4a3eca7f72136.js"></script><br />
And the two commands needed to build and run the container are:<br />
<br />
<script src="https://gist.github.com/horsdal/e3d8f4e2ef4bc74584f8353c66c6b6ca.js"></script><br />
That builds the self-contained app into .\output\release\netcoreapp2.2\linux-x64\publish\<br />
<br />
For context: In my case I needed to build <a href="https://dbup.github.io/">DbUp</a>
as part of the build pipeline for a service that I run in a container. I
want DbUp to be self-contained so I can run it during the deployment
pipeline without needing to install the .NET Core runtime.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-17528946631863490092018-05-06T21:22:00.000+02:002019-10-11T08:37:39.007+02:00Talk: Lightweight Microservice Collaboration using HTTPI had the pleasure of being at the DotNext conference in Sct. Petersburg a couple of weeks ago, and now the video of the talk I did there is out. Enjoy!<br />
<br />
<iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/B-xOKAfIr-M?rel=0" width="560"></iframe>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-46192176557820712792018-01-13T22:18:00.000+01:002018-01-15T08:27:37.561+01:00Event Sourced Aggregates Part 6: Smarter EventsIn this 6th and final post in my <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">series</a> about event sourced aggregates, I will make the events a little bit smarter. So far events have been completely stupid object that just carry some data around, but has no behavior. In this post I give every event a little bit of behavior by moving the 'When' methods from the aggregate to the events. The result is that the aggregate becomes <a href="https://www.martinfowler.com/bliki/AnemicDomainModel.html">anemic</a> (that is, no behavior, just data), does not violate Open/Closed principle anymore, and that the events become more self contained.<br />
<br />
In the<a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-5-anemic.html"> 5th post</a> I made an attempt at moving the 'When' methods out of the aggregate to get to a design where the aggregate does not violate Open/Closed. I did so by introducing a new abstraction - an aggregate projector - but that just ended up with the same Open/Closed violation that the aggregate suffered from originally. Therefore I take another approach in this post.<br />
<br />
<h3>
<span style="color: orange;">Smarter events</span></h3>
<div>
Let's, once again, see how the aggregate looks after domain logic was moved from the aggregate to the command handlers in post 4:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/b6b61a9711d8d558c90e71470363a5ce.js"></script></div>
<br />
And the 'UsernameChangedEvent' looks like this:<br />
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/07fff2e20cb1b0b3fd503fcf49259c68.js"></script></div>
<div>
<br /></div>
<div>
In order to keep the aggregate from growing and growing as features are added to the system, I will move the 'When(UsernameChangedEvent e)` to a `When` method on the event itself, like this:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/985e8b0bcb17772a9cd4c991156cf9e3.js"></script></div>
<div>
<br /></div>
<div>
Now the event holds the data of the event and is responsible for applying the state changes to the aggregate that the event implies. That is: A domain event - like the UsernameChangedEvent - is an indication that something happened to the aggregate, i.e. there was a state change to the aggregate. The 'When' method on the event applies that state change to aggregate that it gets passed in as an argument.</div>
<div>
When moving the 'When' method from the aggregate to the event the signature of the method must change a bit to 'void When(UserAggregate aggregate)'. Notice that this signature is not specific to the 'UsernameChangedEvent', but will be the same for all events. That turns out to be a quite handy side effect of moving the `When` methods. More on that in the next section. Since the `When` signature is the same for all events I'll go ahead and add it to the event interface:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/a9b966153c56005b3c520dcd8b9a3d89.js"></script></div>
<div>
<br /></div>
<div>
Before the 'Event' interface was just a marker interface. Now it shows that all events have a 'When' method.</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">Event replay revisited</span></h3>
<div>
Moving the 'When' methods impact how event replay is done. Remember from the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">first post</a> that what is stored in the database are the events. When we need an aggregate all the events emitted on that aggregate are read from the database, and then the 'When' method for each one is called. Since the 'When' methods each apply the state changes to aggregate implied by the event the end result is an aggregate object in the correct current state of that aggregate. The replay process goes like this:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/0f987ee31357eda66ae350f335f9e5ed.js"></script></div>
<div>
<br /></div>
<div>
Where the event replay is done by the 'Replay' method on the abstract class 'Aggregate'. The special sauce in this is in the implementation of the 'Play' method which - as shown in the first post of the series - involves using reflection to find a suitable 'When' method. This becomes a lot simpler now that all events implement an interface with a 'When' method on it:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/e90954414f32299d1f500477ece02024.js"></script></div>
<div>
<br /></div>
<div>
This simplification was not the goal of the refactoring done in this post, but a nice side effect, and as such an indication that this is indeed a good road to follow.</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">The anemic aggregate</span></h3>
<div>
With the `When` methods moved to the events the aggregate has become anemic. And that is a good thing in this case. The UserAggregate is reduced to this:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/490e24f2245b77a90549059c13d3ec12.js"></script></div>
<div>
<br /></div>
<div>
which is simply a representation of the current state of a user. That is the essence of what the 'UserAggregate' is. Moreover this only changes if we have reason to change how the current state of a user looks, which I consider a good reason to change that class. Moreover the 'UserAggregate' no longer violates the Open/Closed principle since new domain behavior can be added by adding new commands, command handlers and events, but without changing the 'UserAggregate'.</div>
<div>
Often an anemics domain model is seen as an anti pattern in objec oriented code. I agree. But only when looking at the domain model as a whole, not when looking at just one class - like the 'UserAggregate'. My point here is, that looking a the domain model as a whole includes commands, command handlers and events. In that perspective the domain model is not anemic - only the user aggregate is anemic.</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">Wrapping up</span></h3>
<div>
In the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">first</a> post of this series I outlined what I see as a typical C# implementation of event sourced aggregates. I also argued that, that implementation leads to the aggregates violating the Open/Closed principle. In the <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-3-clean.html">third</a> and <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-4-smart.html">fourth</a> posts I solved half of the problem by making the command handlers smarter, and in this post I solved the second half of the problem by making events smarter.</div>
<div>
<br /></div>
<div>
The final code is on <a href="https://github.com/horsdal/event-replay/tree/when-in-events">Github</a>, as are the intermediate steps (see other branches on that same repository).</div>
<div>
<br /></div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-36807992632838364082017-11-22T13:03:00.001+01:002018-01-13T22:23:56.779+01:00Event Sourced Aggregates Part 5: Anemic aggregateIn this 5th part in my series on event sourced aggregates I continue moving code out of the aggregate. In the <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-4-smart.html">last post</a> I moved domain logic out of the aggregate and into the command handlers, making them smart enough to actually handle commands by themselves. In this post I will continue along the path of moving stuff out of the aggregate: I will move the remaining methods, namely the 'When' methods, out from the aggregate to a new abstraction - an aggregate projector. While this will achieve the goal set out in the first post of getting past the aggregate growing and growing over time, the new aggregate projector unfortunately will suffer from the same problem. In the next post I will take another approach moving the 'When' methods out and arrive at a better design, but first let's follow what I think is the most obvious path and see why that leads a bad design.<br />
<br />
<h3>
<span style="color: orange;">The aggregate is a projection</span></h3>
Taking a step back, what is the aggregate? At the datastore level it is a list of events. Together the events represent everything that has happened to the aggregate and, as we have seen, the current state of the aggregate can be recreated from the list of events. At the code level the aggregate is an object - it has some state and it has some behavior. At this point the only behavior it has left is the 'When' methods. The important bit is that the aggregate is an object. It's just an object. Likewise, in the code, different read models are just objects that result from projections over events. In that sense the aggregate is not different from a read model: It is an object that is the result of a projection over events.<br />
<br />
<h3>
<span style="color: orange;">Introducing the aggregate projector</span></h3>
Before I start refactoring lets take a look at how the aggregate looks right now:<br />
<br />
<script src="https://gist.github.com/horsdal/b6b61a9711d8d558c90e71470363a5ce.js"></script><br />
<br />
The aggregate has some state represented by the properties on lines 3 and 4, and then some 'When' methods that make up the logic needed to perform the projection from the aggregates events to its current state.<br />
<br />
Seeing that a new 'When' method will be added to the aggregate every time a new event is introduced - and new features will usually result in new events, in my experience - the aggregate still has the problem of growing big and unwieldy over time. So let's introduce another class that can do the projections:<br />
<br />
<script src="https://gist.github.com/horsdal/56e117f01ed65c1d34468d8065ebf34c.js"></script><br />
<br />
This doesn't just work. First off the new 'UserAggregateProjector' cannot set the properties on the aggregate to anything. That can be fixed by adding internal setters to the aggregate, allowing the projector to access the setters, but disallowing access from outside the same project as the 'UserAggregate', which I expect to mean anything beyond commands, command handlers and events.<br />
Furthermore the event replay done when fetching an aggregate must also change from calling 'When' methods on the aggregate to calling them on the 'UserAggregateProjector'. That means changing 'Aggregate' base class to this:<br />
<br />
<script src="https://gist.github.com/horsdal/49ac3266b2261a5ec019b58143479132.js"></script><br />
<br />
The changes are the introduction of the 'GetProjector' method on line 30 and the use of that new method in the 'Play' method, which now does reflection of the projector class to find the 'When' methods instead of doing it over the aggregate. The end result is the same: An aggregate object with the current state of the aggregate recreated by replaying all events.<br />
<br />
Moving the 'When' methods has obviously also changed the aggregate, which now only contains state:<br />
<br />
<script src="https://gist.github.com/horsdal/e45beab8f4e4db6109d49ae5575146c9.js"></script><br />
<br />
This is what is known as an <a href="https://www.martinfowler.com/bliki/AnemicDomainModel.html">anemic domain model</a>, because it has no behavior. That's usually considered an anti-pattern, but I don't necessarily agree that it is; as argued above the aggregate is essentially a projection of the events, so I do not see why that object has to be where the domain behavior goes. As we saw in the 4th post of the series command handlers is a nice place to put domain behavior. <br />
<br />
<h3>
<span style="color: orange;">The projector violates Open/Closed principle</span></h3>
<br />
As a stated at the beginning of this post the design I've arrived at now is not good: The new 'UserAggregateProjector' suffers just as much from perpetual growth as the aggregate did before I moved the 'When' methods out of it. In other words the new projector violates the Open/Closed principle, which is what I am trying to get away from. So I have not solved anything, just moved the problem to a new abstraction :( Seems like I need to take another iteration, which I will in the <a href="http://www.horsdal-consult.dk/2018/01/event-sourced-aggregates-part-6-smarter.html">next post</a>.<br />
<br />
The code for this post is in <a href="https://github.com/horsdal/event-replay/tree/anemic-aggregate">this branch</a>.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-71279900914934933472017-11-14T07:19:00.000+01:002017-11-14T07:19:15.212+01:00Event Sourced Aggregates Part 4: Smart command handlersIn this fourth post in my <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">series</a> about event sourced aggregates I will continue work on the command handlers. In the <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-3-clean.html">3rd post</a> I did some cleaning up of the command handlers, cutting them down to very little code. In this post I will make the command handlers smart by moving domain logic from the aggregate to the command handlers.<br />
<br />
<h3>
<span style="color: orange;">Motivation</span></h3>
<div>
In the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">first</a> and <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-2-where.html">second</a> posts of the series I outlined a typical C# implementation of event sourced aggregates and how that style of implementation leads to ever growing aggregates - every added features adds (at least) two methods to the aggregate: </div>
<div>
<ul>
<li>One for the domain logic. E.g. a 'ChangeUsername' method, that has whatever business logic there is around changing the username. If and when these methods decide a change to the aggregate state is needed they emit a domain event.</li>
<li>A 'When' method for any new events. The `When` methods perform all state changes on the aggregate.</li>
</ul>
<div>
The patterns I see in the implementations I come across is that there is a one-to-one correspondence between commands, command handlers and public domain logic methods on the aggregate. For example the pattern is that for a 'ChangeUsernameCommand' there is a 'ChangeUsernameCommandHandler' class and a 'ChangeUsername' method on the aggregate. I took almost all the plumbing out of the command handler in the last post and essentially left it at this:<br />
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/85977df4af26de56329aca1d062f57db.js"></script><br /></div>
<div>
which invokes the 'helper.Handle' method to get all the plumbing done and then calls the 'ChangeUsername' method to get the domain logic done. So in essence the command handler just delegates to the aggregate, but isn't the responsibility of the command handler to ... handle the command? I think it is. And handling the command means running the domain logic, so let's move that logic from the aggregate to the command handler.<br />
<br />
<h3>
<span style="color: orange;">Smart command handlers</span></h3>
In the second post I introduced the 'ChangeUsernameCommand` and the associated command handler and methods on the aggregate. In particular this `ChangeUsername` method on the aggregate:<br />
<br />
<script src="https://gist.github.com/horsdal/6e1d9b234d20e9bb331c554742ea17ca.js"></script><br />
<br />
which implements the domain logic for changing username. That is the logic I want to move to the command handler.<br />
Moving the domain logic straight over the command handler, changes the `Handle` method on the command handler to this:<br />
<br />
<script src="https://gist.github.com/horsdal/339b6d621cc03dd85ff1864155b0e56e.js"></script><br />
<br />
Now the command handler contains the logic for handling the command. Note that now the command handler also emits domain events - on line 7. This makes sense since this is still event sourced, so changes to the aggregate state are still done through events. The rest of the mechanics around events remain unchanged: The `Emit` method on the base aggregate still calls the 'When' method for the event and stores the event to the list of new events on the aggregate. Saving the aggregate still means appending the list of new events to the event store, and getting the aggregate from the 'AggregateRepository' still means reading all the aggregates events from the event store and replaying each one.<br />
<br />
Having moved the domain logic out of the aggregate I have a slimmer aggregate, that only has the state of the aggregate and the 'When' methods. In the next two post I slim down the aggregate even further by moving the 'When' methods out.<br />
<br />
The complete code for this post in <a href="https://github.com/horsdal/event-replay/tree/smart-command-handlers">this branch</a>.</div>
</div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-64610497114475433482017-11-06T23:14:00.000+01:002018-01-13T22:22:27.058+01:00Event Sourced Aggregates Part 3: Clean up command handlersThis is the 3rd part of my series about event sourced aggregates. In the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">first post</a> I outlined a typical implementation of event sourced aggregates, and in the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-2-where.html">second post</a> I showed how that implementation leads to aggregates that grow bigger and bigger as features are added to the system. In this post I will clean up the command handlers from the previous post by moving some repeated code out them to a new abstraction. In the coming posts I will refactor between the command handlers, aggregates and event handlers to arrive at a design where the aggregate does not grow.<br />
<br />
<h3>
<span style="color: orange;">Repeated code in command handlers</span></h3>
<div>
In the last post we looked briefly at this implementation of a command handler:</div>
<div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/cf69c8425837f3a1dde4d069c6f5b897.js"></script></div>
</div>
<div>
<br /></div>
<div>
Looking at the above the code at line 20 is - in a sense - where the ChangeUsernameCommand the is handled, because that is the only line that is about changing a username. All the other code in the command handler is about infrastructure; loading the aggregate, saving the aggregate and dispatching events. Moreover the code for loading and saving aggregates as well as the code for dispatching will be repeated in every command handler</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">Introducing a helper</span></h3>
<div>
To get past that repetitiveness and to cut back on the amount of infrastructure code in the command handler, we introduce this helper, where the loading of the aggregate, the saving of the aggregate and the dispatching of events is done:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/b7ff8133c70ee456b9c5ab1c277ea00e.js"></script></div>
<div>
<br /></div>
<div>
The idea behind the CommandHandlerHelper is that the concrete command handler calls the Handle method with a handlerFunc, that does the business logic bit of the command handler. The handlerFunc is called at line 18, so the helper makes sure the infrastructure code is done in right order in relation to the business logic.</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">Cleaner command handlers</span></h3>
<div>
With the CommandHandlerHelper in place the ChangeUsernameCommand can be rewritten to use it like this:</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/horsdal/1e6314b410b21956ab170d8759e8e8be.js"></script></div>
<div>
This is a good deal simpler than the command handler code at the start of the post.</div>
<div>
<br />
<br /></div>
<div>
That's it for now. With this clean up in place we set for the next steps:</div>
<div>
<ul>
<li>Make the <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-4-smart.html">command handlers smarter</a></li>
<li>Make the aggregate anemic in a <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-5-anemic.html">naive way</a>, leaving a stable aggregate, but introducing a new Open/Closed violation</li>
<li><a href="http://www.horsdal-consult.dk/2018/01/event-sourced-aggregates-part-6-smarter.html">Make the aggregate anemic, events (a little) smart, and solve the Open/Closed violation</a></li>
</ul>
<div>
</div>
</div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-42919349742612983712017-10-31T23:07:00.000+01:002018-01-13T22:20:57.882+01:00Event Sourced Aggregates Part 2: Where the mess startsIn the <a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-1-outline.html">first post</a> in this series I outlined a typical C# implmentation of event sourced aggregates. In this post we add a second feature to the example from the first post. In doing so I wish to illustrate how that typical implmentation leads to violation of <a href="https://en.wikipedia.org/wiki/Open/closed_principle">Open/Closed principle</a> and ever growing aggregates.<br />
<br />
<h3>
<span style="color: orange;">The second feature</span></h3>
<div>
Once we have the code from the first post in place - that is: The infrastructure for sending commands to aggregates, raising events from aggregates, saving events and replaying events - and need to add a second feature, the way to do it is (to some extent) as outline in the first post: </div>
<div>
<ol>
<li>Send a new type of command</li>
<li>Implement a command handler for the new command</li>
<li>Implement a new method on the aggregate with the new domain logic</li>
<li>Emit any new events needed</li>
<li>Implement new When methods for any new event types</li>
</ol>
<div>
Let's say we want to be able change a users username.</div>
</div>
<div>
The command and the sending of of that command looks like this:<br />
<br />
<script src="https://gist.github.com/horsdal/1c4c515637a6da5eeef0526b552565d9.js"></script><br />
<br />
That's pretty straight forward and not too interesting, so let's move on to the command handler:<br />
<br />
<script src="https://gist.github.com/horsdal/cf69c8425837f3a1dde4d069c6f5b897.js"></script><br />
<br />
That's also pretty straight forward, but a little more interesting: Most of the the command handler code is something that will be repeated in all commands handlers. We will deal with this in the next post, where that repetition is pulled out into a helper class.<br />
The next step is changes to the aggregate, where this is added:<br />
<br />
<script src="https://gist.github.com/horsdal/6e1d9b234d20e9bb331c554742ea17ca.js"></script><br />
<br />
This is still pretty straight forward. In fact everything needed to add this new feature was straight forward, so that is a good thing. The problem lies in the last two methods, the ones added to the aggregate.</div>
<div>
<br /></div>
<h3>
<span style="color: orange;">Problem: An Open/Closed Principle violation</span></h3>
<div>
The aggregate just grew. In order to support changing the username we added 2 methods to the aggregate. That violates the Open/Closed principle, which indicates that it is a potential problem. In my experience, it quickly becomes a real problem because the aggregate grows relatively quickly and eventually becomes big and complicated, just like any other class that grows monotonically.<br />
<br />
That's it for now. The next posts will:<br />
<ol>
<li>Make the command handlers <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-4-smart.html">smarter</a> and <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-3-clean.html">clean up</a> some repetitiveness</li>
<li>Make the aggregate anemic in a <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-5-anemic.html">naive way</a>, leaving a stable aggregate, but introducing a new Open/Closed violation</li>
<li><a href="http://www.horsdal-consult.dk/2018/01/event-sourced-aggregates-part-6-smarter.html">Make the aggregate anemic, events (a little) smart, and solve the Open/Closed violation</a></li>
</ol>
</div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-28667326293492877502017-10-24T12:01:00.000+02:002018-01-13T22:19:26.490+01:00Event Sourced Aggregates Part 1: Outline of a typical implementationThis is the first post in a series of posts that takes its offset in a design problem I've encountered repeatedly with event sourced aggregates: They grow every time a feature is added. Nothing (almost) is removed from them, so over time they grow very big and gnarly. Why does this happen? Because typical implementations of event sourced aggregates violate the <a href="https://en.wikipedia.org/wiki/Open/closed_principle">Open/Closed principle</a>.<br />
Through this series of post, I will show how event sourced aggregates violate Open/Closed and - as a consequence - tend to grow monotonically, and then show how we can address that by re-factoring away from the big aggregate towards a small and manageable one. Cliffhanger and spoiler: The aggregate will become anemic and I think that is a good thing.<br />
<br />
The complete code from the series is on <a href="https://github.com/horsdal/event-replay">GitHub</a>, where there is a <a href="https://github.com/horsdal/event-replay/tree/second-event">branch </a>with the code for the first two posts.<br />
<br />
<h3>
<span style="color: orange;">Event sourced aggregates</span></h3>
<div>
The context of what I am exploring in this series of posts is systems based on <a href="https://www.amazon.com/exec/obidos/ASIN/0321125215/domainlanguag-20">Domain Driven Design</a>, where some or all of the <a href="https://martinfowler.com/bliki/DDD_Aggregate.html">aggregates </a>are stored using <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing">event sourcing</a>. Often these systems also use <a href="https://martinfowler.com/bliki/CQRS.html">CQRS</a> - very much inspired by Greg Youngs <a href="https://www.youtube.com/watch?v=JHGkaShoyNs">talks</a> and <a href="https://cqrs.files.wordpress.com/2010/11/cqrs_documents.pdf">writings</a>.<br />
Using event sourcing for storing the aggregates, means that the aggregate code does not change the state of the aggregate directly, instead it emits an event. The event is <i>applied</i> to the aggregate which is where changes to the state of the aggregate happens, but the event is also stored to a data store. Since aggregate state is only changed when an event is applied, the current aggregate state can be recreated by reading all the events for a given aggregate up form the data store and applying each one to the aggregate. The benefits of this approach are many (when applied to a suitable domain) and described elsewhere, so I wont go into them here.<br />
<br /></div>
<h3>
<span style="color: orange;">A typical C# implementation</span></h3>
<div>
Typically implementations of all this follow a structure where requests coming in from the outside - be it though a client making a request to an HTTP endpoint, a message on a queue from some other service or something else - result in a chain that goes like this:</div>
<div>
<ol>
<li>A command is sent, asking the domain to perform a certain task </li>
<li>A command handler picks up that command, fetches the appropriate aggregate and triggers appropriate domain behavior. Fetching the aggregate involves replaying all the aggregates events (event sourcing!)</li>
<li>A method on an aggregate is called, and that is where the actual domain logic is implemented</li>
<li>Whenever the domain logic needs to change some state it emits an event (event sourcing, again) of a specific type</li>
<li>A 'when' method for the specific event type on the aggregate is called and updates the state of the aggregate</li>
<li>After the aggregate is done, all the events emitted during execution of the domain logic is dispatched to any number of event handlers that cause side effects like updating view models, or sending messages to other services.</li>
</ol>
<div>
To put this whole process into code, let's think of an example: Creating a user in some imaginary system. The first step is to send the create user command:</div>
</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/54ef025988da65672e18b9a39d2dad74.js"></script><br />
<div>
Next step is the event handler for the create user command. Note that in this example I use the <a href="https://github.com/jbogard/MediatR">MediatR </a>library to connect the sending of a command to the handler for the command.</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/a563048447558a9b308ed33173ab88fa.js"></script><br />
<div>
Note that most of what is going on here is the same as for other handlers for other commands: Pick the right aggregate and, after executing domain logic, save that aggregate and then dispatch whatever events were emitted.</div>
<div>
<br /></div>
<div>
On line 19 of the handler we call into the aggregate. The code in the aggregate looks like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/40b727bf5f67c715c220757ad580fe46.js"></script><br />
<div>
At line 11 we call the Emit method. This is how most implementations I've seen work, and typically that Emit method is part of the Aggregate base class and looks something like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/36e416fb091eac5a4f5340e449731018.js"></script><br />
<div>
Notice how Emit calls Play which uses reflection to find a When method on the aggregate itself, and to call that When method. The When method is supposed to update the state of the aggregate and is also the method that gets called during event replay. More on that below. For now let's see the when method:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/06a867e061704f8f3d68488d9b34d5a5.js"></script><br />
<div>
That's pretty much it, though there a few things I have skipped over a bit quickly: How the aggregate is fetched, how it is saved and how events are dispatched. I will not go into the event dispatching, since it is not relevant to the point I am making in this series, but the code is on <a href="https://github.com/horsdal/event-replay/blob/second-event/EventReplay/Infrastructure/EventDispatcher.cs">Github</a>, if you want to look. As for the other two bits - fetching and saving aggregates - lets start with how aggregates are saved:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/344f9603f5316956218931556e470dd1.js"></script><br />
<div>
As you can see saving the aggregate essentially means saving a list of events. The list should contain all the events that has ever been emitted by the aggregate. That is the essence of event sourcing. When it comes to fetching the aggregate, the list of events is read, and each one is replayed on a new clean aggregate object - that is the When methods for each event is called in turn. Since only the When methods update the state of the aggregate the result is an aggregate object in the right state. The Get method on the aggregate repository (which does the fetching) looks like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/442d77d754cb0c25eb6b102b4707052e.js"></script><br />
<div>
And the Replay method called in line 14 just runs through the list of events and plays each on them in turn, like this:</div>
<div>
<br /></div>
<script src="https://gist.github.com/horsdal/0f987ee31357eda66ae350f335f9e5ed.js"></script><br />
<div>
That pretty much outline the implementations of event sourced aggregates I seem to come across. </div>
<div>
<br /></div>
<div>
That's it for now. The next posts will:<br />
<br />
<ol>
<li><a href="http://www.horsdal-consult.dk/2017/10/event-sourced-aggregates-part-2-where.html">Add a second feature and see how the aggregate starts to violate Open/Closed principle</a></li>
<li>Make the command handlers <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-4-smart.html">smarter</a> and <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-3-clean.html">clean up</a> some repetitiveness</li>
<li>Make the aggregate <a href="http://www.horsdal-consult.dk/2017/11/event-sourced-aggregates-part-5-anemic.html">anemic in a naive way</a>, leaving a stable aggregate, but introducing a new Open/Closed violation</li>
<li><a href="http://www.horsdal-consult.dk/2018/01/event-sourced-aggregates-part-6-smarter.html">Make the aggregate anemic, events (a little) smart, and solve the Open/Closed violation</a></li>
</ol>
</div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-29872936044008944982017-08-29T23:25:00.002+02:002017-08-30T08:58:26.729+02:00Free microservices ebookIn the early summer I helped Manning put together a short, free ebook about microservices. It's made up of chapters from other Manning books and as such it doubles as a sampler or appetizer for those books and as a short quick intro to some important topics relating to microservices.<br />
<br />
You can get from <a href="http://bit.ly/2xwMPfV">here</a>.<br />
<br />
The books the chapters are from are:<br />
<br />
<ul>
<li><a href="https://manning.com/books/microservices-in-net-core?a_aid=horsdalconsult&a_bid=66b54382">My own microservices book</a></li>
<li><a href="https://www.manning.com/books/reactive-applications-with-akka-net">Reactive Applications with Akka.NET</a> by Anthony Brown</li>
<li><a href="https://www.manning.com/books/the-tao-of-microservices">The Tao of Microservices</a> by Richard Rodger</li>
<li><a href="https://www.manning.com/books/docker-in-action">Docker in Action</a> by Jeff Nickoloff</li>
</ul>
<div>
Enjoy!</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://bit.ly/2xwMPfV"><img alt="" border="0" data-original-height="800" data-original-width="636" height="320" src="https://images.manning.com/720/960/resize/book/f/61c0c7b-4bb7-43fb-b327-49110e2acbbd/Gammelgaard_hires.png" title="Exploring microservices cover" width="254" /></a></div>
<div>
<br /></div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-51769089061839576662017-03-28T14:21:00.000+02:002019-10-11T08:37:58.124+02:00Talk video: "Consolidating Services with middleware" from NDC London 2017Back in January I did a talk called at <a href="http://ndc-london.com/">NDC London</a> called "Consolidating Services with middleware", and the video of the talk has come out. Enjoy!<br />
<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="360" mozallowfullscreen="" src="https://player.vimeo.com/video/209648860" webkitallowfullscreen="" width="640"></iframe><br />
<a href="https://vimeo.com/209648860">Consolidating Services with middleware - Christian Horsdal</a> from <a href="https://vimeo.com/ndcconferences">NDC Conferences</a> on <a href="https://vimeo.com/">Vimeo</a>.<br />
<br />
In case you are wondering what the talk is about, here is the abstract:<br />
<blockquote class="tr_bq">"Have many services? Writing new ones often? If so middleware can help you cut down on the ceremony for writting new services and at same time consolidate the handling of cross cutting concerns. But what is middleware? OWIN and ASP.NET Core both have a concept of middleware. What are they? How do they help? In this talk we will dive into the code, write some middleware and show how middleware helps you handle cross-cutting concerns in an isolated and re-usable way across your services. I'll compare and contrast the OWIN and ASP.NET Core middleware concepts and talk about where each is appropriate."</blockquote>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-41771046939847125632017-02-12T10:59:00.000+01:002017-02-12T10:59:19.844+01:00Event-based Collaboration does not imply Event Sourcing<h2>
Abstract</h2>
<div>
In my <a href="https://www.manning.com/books/microservices-in-net-core">microservices book</a> I talk about three styles of collaboration between microservices: Command, Query and Event based collaboration. On the <a href="https://forums.manning.com/forums/microservices-in-net-core">book's forum</a> I recently got <a href="https://forums.manning.com/posts/list/39956.page">a question</a> that boils down to "if a microservice allows event-based collaboration, can all it's state always be recreated from those events?". I'm taking the liberty of re-framing this question as "if a microservice allows event-based collaboration, does it have to use <a href="http://docs.geteventstore.com/introduction/4.0.0/event-sourcing-basics/">Event Sourcing</a>?" My answer is: No - you can, but you don't have to.</div>
<div>
I will try to explain why below.<br />
<br /></div>
<h2>
Event Based Collaboration</h2>
<div>
When we use a microservice architecture for a system, that system gets broken into lots - usually hundreds - of small narrowly focused services each of which handle one specific capability. In order to deliver a cohesive experience to the users using the applications built on top of all those microservices we have to make them collaborate - the alternative would be to expose the end user to that very fine grained break down into single capability services. That would make for an insane user experience.<br />
<br />
One of the ways microservices can collaborate is through events: When something significant happens in a microservice it can choose to publish an event that other microservices can then react to however they need/wish to.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFjJZgP6l8W88ImgJeSGHfafQExuJuWNpF70oQDMtXm794Yq0S02gdN6MsXEG0IR235HI_GzSXJjiPgT5jNLc0-SiPCnWTJBcTk9j86f3Q_vnedSi8jcrYqeL8QtF82CtgNz3_ZWb0IjU/s1600/external+events.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFjJZgP6l8W88ImgJeSGHfafQExuJuWNpF70oQDMtXm794Yq0S02gdN6MsXEG0IR235HI_GzSXJjiPgT5jNLc0-SiPCnWTJBcTk9j86f3Q_vnedSi8jcrYqeL8QtF82CtgNz3_ZWb0IjU/s640/external+events.png" width="640" /></a></div>
<br />
This is a powerful style of collaboration. Events allow for asynchronuous processing, for slow subscriberss to catch up with bursts at their own pace, for some microservices to be down for shorter periods and more. For these reasons event-based collaboration between microservices is quite often a better choice than command- or query-based collaboration. The point in this context, though, is more about what the events are. These events are things that are significant outside of the microservice publishing them. They are published in order to drive collaboration with other microserivces.<br />
<br /></div>
<h2>
Internal Event and External Events</h2>
<div>
The events published to other microservices in order to drive collaboration are events that can - and should - be published regardless of how the publishing microservice is implemented. Referring to the figure above: The blue microservice publishes events to drive collaboration with other microservices, like the green and yellow ones. That has nothing to do with the inner workings of the blue microservice. The events published to other microservices are <i>external</i> events.<br />
The blue microservice from the figure above can store its state however it wants. One of the ways it can chose to store its state is using <a href="http://docs.geteventstore.com/introduction/4.0.0/event-sourcing-basics/">Event Sourcing</a>. Using Event Sourcing introduces another set of events: Ones stored internally in the microservice. These events to do not drive collaboration with other microservices, but they capture every little state change in the blue microservice.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLzWdnxUpGic9kVJhMUyc-mM2wWGlnqTLPIpslLJ0QVDMje-p-56C56qwn4X4jFxMnFU4KgSUWh8hfNwm63KIXdrk2RjfJRNuGzhKpFciEKeSF5Ipjhkmg1N6ybAXcKG4EaCBCb2lg11k/s1600/internal+and+external+events.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLzWdnxUpGic9kVJhMUyc-mM2wWGlnqTLPIpslLJ0QVDMje-p-56C56qwn4X4jFxMnFU4KgSUWh8hfNwm63KIXdrk2RjfJRNuGzhKpFciEKeSF5Ipjhkmg1N6ybAXcKG4EaCBCb2lg11k/s400/internal+and+external+events.png" width="400" /></a></div>
<br />
I find it helpful to distinguish between these two types of events:<br />
<br />
<ul>
<li><i>External events</i> drive collaboration with other microservices. These are also sometimes referred to as integration events.</li>
<li><i>Internal events</i> captures all state changes within the microservices and is the basis for event sourcing. These are also sometimes referred to as domain events.</li>
</ul>
<div>
Internal events are usually far more granular than external events. Internal events are also tightly coupled to the implementation details of the microservice, whereas external events are not.</div>
</div>
<h2>
</h2>
<h2>
Conclusion</h2>
<div>
Having established the difference between internal and external events it should be clear that one does not imply the other. A microservice can publish external events without using Event Sourcing. Likewise a microservice can use Event Sourcing without publishing any external events.</div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-20226472184150982532017-01-29T09:06:00.000+01:002017-01-29T22:34:03.856+01:00"Microservices in .NET Core" has landedMy new book "<a href="http://manning.com/books/microservices-in-net-core?a_aid=horsdalconsult&a_bid=66b54382">Microservices in .NET Core - with examples in Nancy</a>" has been published. It's been available as an early access ebook for a long time, but now the final paper book and the final ebook is available.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9QAWDTu897oEK2HYHTiG013vss0WdKbuGdUqqGp0kUmz_sUHoQ2M47lV_rcyUFqcuVQ7n_kdbQ6cP4JrWJrORZFVGVNAkmOJ-MRSlv_3VLk3H77TxMkjeGQ-rATt-LZXFpmptEmn8cIA/s1600/Horsdal-Microservices-HI.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9QAWDTu897oEK2HYHTiG013vss0WdKbuGdUqqGp0kUmz_sUHoQ2M47lV_rcyUFqcuVQ7n_kdbQ6cP4JrWJrORZFVGVNAkmOJ-MRSlv_3VLk3H77TxMkjeGQ-rATt-LZXFpmptEmn8cIA/s1600/Horsdal-Microservices-HI.png" title="Microservices in .NET Core" /></a></div>
<br />
This book is very much based on experience from working of real microservices systems. While the examples in the book are not lifted directly from any particular system they are inspired by real world systems. My rule has been that all the examples demonstrate something that could have been in one or more of the systems I have experience with.<br />
<br />
With this book I have tried to show two things:<br />
<br />
1. How to design a microservices system.<br />
2. How to build such a system using a lightweight .NET based tech stack.<br />
<div>
<br /></div>
<div>
All but one chapter reflect these two levels, in that they start off explaining an aspect of designing a microservice system, and then go on to show how to implement it using my .NET tech stack of choice. This should help make clear which parts make sense regardless of tech stack and which are tech specific. At the end of the book you should have the tools to both design a microservice system and implement it, as we have covered subjects like how to decide what belongs in which microservice, where data resides, how microservices collaborate, how to test such a system, how to make it robust and how gain insights into how it is doing in production.</div>
<div>
<br /></div>
<div>
The tech stack I use in the book is based on .NET Core, Microsoft's new cross platform version of .NET. On top of .NET Core I use the <a href="http://nancyfx.org/">Nancy web framework</a> and <a href="http://owin.org/">OWIN</a> middleware, which forms a powerful yet simple stack. To compliment this I use various other OSS libraries where appropriate, like <a href="http://www.thepollyproject.org/">Polly</a> and <a href="https://github.com/StackExchange/dapper-dot-net">Dapper</a>. The guiding principle in all the tech choices is that they should be simple to use - make the simple stuff simple, while also allowing to do more complex stuff without unwarranted hassle.</div>
<div>
<br /></div>
<div>
You can find the table of contents and a couple of sample chapter on the <a href="http://manning.com/books/microservices-in-net-core?a_aid=horsdalconsult&a_bid=66b54382">books page at Manning</a>. </div>
<div>
Also, if you are interested in these things, I have a <a href="https://kursusportal.it/undervisning/beskrivelse/LB3210/.NET+-+Microservices+i+.NET+-+Foundation">3 day course</a> around these things available.</div>
<div>
<br /></div>
Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-57251598870634945142016-11-07T21:04:00.000+01:002016-11-07T21:04:00.363+01:00Sharing and Caching Assets Across Single-Page BFFsOver the last couple of posts I showed the <a href="http://www.horsdal-consult.dk/2016/09/single-page-bffs.html">single-page BFFs pattern</a> (a variation of the Backend-For-Frontend aka BFF pattern where the BFF serves only one page in a web application), and how you will likely need to place <a href="http://www.horsdal-consult.dk/2016/10/using-single-page-bffs-and-hiding-it.html">single-page BFFs behind a reverse proxy</a>.<br />
<br />
In this post I will outline an approach to handling shared assets - like CSS or Javascript which is used on all or most of the pages in a web application built using a number of single-page BFFs.<br />
<br />
When building a web application consisting of a number of pages, there is likely both some look-and-feel and some behavior that we want to share across all pages - buttons should look and behave the same, menus that should be on all pages, colors and fonts should be consistent etc. This calls for sharing some CSS and some Javascript across the pages. When thinking about handling these shared things across a number of single-page BFFs there are some competing requirements to take into consideration: <br />
<br />
<ul>
<li>Clients - i.e. browsers - should be able to cache the CSS and Javascript as the user moves through the web application. The alternative is that the client downloads the same CSS and JS over and over as the user navigates the web application. That would be too wasteful in most cases.</li>
<li>Each single-page BFF should be individually deployable. That is we should be able deploy a new version of one single-page BFF without having to deploy any other services. If not the system soon becomes unmanageable. Furthermore developers should be able to run and test each single-page BFF by itself without having to spin up anything else.</li>
<li>I do not want to have to write the same CSS and JS in each single-page BFF. I want to share it.</li>
</ul>
<br />
In this post I will show an approach that allows me to achieve all three of the above.<br />
<br />
First notice that the obvious solution to the first bullet - allowing clients to cache shared CSS and JS - is to put the shared CSS and the shared JS in two bundles - a CSS bundle and a JS bundle - and put the bundles at a known place. That way each single-page BFF can just address the bundle at the known place, as shown below. Since all pages refer to the same bundles the client can cache them.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxlDEBUF0t0sktzs5kva2iAg3HGxCFaa5Bd9oG8hZ8645agABd9sGQUepdioVQuXOUnGN4EY6rb4trlLRDT7fBJKGW__t5NEDjmi8Sv9W2UNTWCtzbdqv5O9l66lx_Lj4_SzGv5R14Cwo/s1600/single-page-bff-with-shared-assets.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="326" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxlDEBUF0t0sktzs5kva2iAg3HGxCFaa5Bd9oG8hZ8645agABd9sGQUepdioVQuXOUnGN4EY6rb4trlLRDT7fBJKGW__t5NEDjmi8Sv9W2UNTWCtzbdqv5O9l66lx_Lj4_SzGv5R14Cwo/s640/single-page-bff-with-shared-assets.png" width="640" /></a></div>
<br />
The problem with this is that it violates bullet 3: Any given single-page BFF only works when both the global bundles are available, so for a developer to work on a single-page BFF, they need not only the BFF, but also something to serve the bundles. If we are not careful, this approach can also violate bullet 2: Since the single-page BFFs depend on the two global bundles, but they do not contain the global bundles themselves, each one has a hard dependency on another service - the one that serves the bundles. That means we can easily end up having to deploy the service that serves the bundles at the same time as a new version of one of the single-page BFFs.<br />
<br />
Looking instead a bullet three first suggests that we should put the shared CSS and Javascript into a package - an NPM package, that is - and include that package in each single-page BFF. That means each single-page BFF has the bundles it needs, so developers can work with each single-page BFF in isolation. This also means that each single-page BFF continues to be individually deployable: When deployed they bring along the bundles they need, so there is no dependency on another service to serve the bundles. The problem now becomes how to allow clients to cache the bundles? When the bundle are included in each single-page BFF, each page will use different URsL for the bundles - completely defeating HTTP cache header.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaucyu-910nnSKE1F6pd5RbqyflVZJKbwizbjzVxv3ylF0Mdk2D8G3i65yQoHslY9gVAnLooKQ8Lvlyt4W4Fga7iNzq12ejhhyQhAsygLmDBNx6AuRdlrzZShEL0E3wNR7ITTAnR3AUhc/s1600/single-page-bff-with-shared-npm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="410" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaucyu-910nnSKE1F6pd5RbqyflVZJKbwizbjzVxv3ylF0Mdk2D8G3i65yQoHslY9gVAnLooKQ8Lvlyt4W4Fga7iNzq12ejhhyQhAsygLmDBNx6AuRdlrzZShEL0E3wNR7ITTAnR3AUhc/s640/single-page-bff-with-shared-npm.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Luckily, as discussed in the last post, the single-page BFFs are already behind a reverse proxy. If we put a bit of smarts into the reverse proxy we can include the bundles in each single-page BFF and still let every page use same the URLs for the bundles, like so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGacI3fbuFUn6iWuvLyI9kimEFpPOLUlEEB-3SzimBvSKpNofXA4bNso1Dyfs1tqclgoSvHLcXis2p2vh-jDYrAu4j1bQuUNTWgzgHWM1zj9BRsGaTdYj2JhwopuC_mticm0aNzvZYn40/s1600/single-page-bff-with-package-and-proxy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="448" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGacI3fbuFUn6iWuvLyI9kimEFpPOLUlEEB-3SzimBvSKpNofXA4bNso1Dyfs1tqclgoSvHLcXis2p2vh-jDYrAu4j1bQuUNTWgzgHWM1zj9BRsGaTdYj2JhwopuC_mticm0aNzvZYn40/s640/single-page-bff-with-package-and-proxy.png" width="640" /></a></div>
<br />
The trick implementing this is to have the reverse proxy look at the referrer of any request for one of the bundles - if the referrer is frontpage, the request is proxied to the frontpage single-page BFF, if the referrer is the my account page, the request is proxied to the MyAccount single-page BFF. That means that the bundles loaded on the frontpage is the bundles deployed in the frontpage single page BFF. Likewise the bundles loaded on the my account page come from the MyAccount single-page BFF. But from the client perspective the bundles are on the same URL - i.e. the client is able to cache the bundles based on HTTP cache headers.<br />
<br />
To sum up: Put shared CSS and JS bundles in an NPM package that all the single-page BFFs use, and making the reverse proxy use the referrer on requests for the bundles to decide where to proxy requests to allows clients the cache the bundles, the single-page BFFs to be individually deployable and developers work with each single-page BFF in isolation.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-2787616367072019552016-10-05T23:06:00.000+02:002016-10-05T23:06:35.698+02:00Using Single-Page BFFs and Hiding ItIn the <a href="http://www.horsdal-consult.dk/2016/09/single-page-bffs.html">last post</a> I wrote about a variation of the <a href="http://samnewman.io/patterns/architectural/bff/">BFF - Backend-for-frontend - pattern</a>: The single-page BFF page, where the BFF does not support a full web frontend, but only a single page. The reason to use this pattern is that even BFFs can grow out of hand, at which point we can consider cutting the BFF into several single-page BFFs, each responsible for supporting just on page.<br />
<br />
Now imagine we have a web site that uses the single-page BFF patterns extensively. As users navigate through the site and visit different pages they will in effect interact with different single-page BFFs. Each such single-page BFF is its own service, and conseqeuntly has its own host name - e.g. the frontpage BFF might be at https://frontpage.bff.ajax.com, the "my account" page might be at https://myaccount.bff.ajax.com/ and so on for each logical page on the site. Like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFruH0V0VM2dxAvIsDv6OZzMKwqk1LaasuJvvzfgnmRYJhIujjMRVpDbl_Z0hevey8T25fDSFDAAE-oaT1b19S_T6MWK00G_ZEMiFEfanWxRfajIrZrR-gecI_UK8LXUgfQVTfgiORlkU/s1600/three-single-page-bffs.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFruH0V0VM2dxAvIsDv6OZzMKwqk1LaasuJvvzfgnmRYJhIujjMRVpDbl_Z0hevey8T25fDSFDAAE-oaT1b19S_T6MWK00G_ZEMiFEfanWxRfajIrZrR-gecI_UK8LXUgfQVTfgiORlkU/s640/three-single-page-bffs.png" width="640" /></a></div>
<br />
The problem we now have is that if users access these adresses - https://frontpage.bff.ajax.com, https://myaccount.bff.ajax.com etc. - we are exposing the way we have chosen to implement the system on the server side to a our users. That's bad: Users will bookmark these and then we are tied to keeping these host names alive, even if we decide to rearchitect the server side to use some other pattern than single-page BFFs. Instead I would like users to just stay of the same domain all the time - like https://www.ajax.com/ - and just visit different pages on that domain - e.g. https://www.ajax.com/ and https:/www.ajax.com/account. So how do we make ends meet? We simply set up a reverse proxy in front of the single-BFFs. Like so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZaAjdHgKSnS-TncwohiBkjHHTr-CoggC2Enmdbu_sptTMZFw3b-suM5YzzjVTmSLu3QByLPD0FAsfIAmlKQlNRV2RVh-D_Rmz_ygRpaC6bLeyJSyygearE2qYPrYLIg_YRl_6nEK50cs/s1600/three-single-page-bffs-behind-proxy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="448" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZaAjdHgKSnS-TncwohiBkjHHTr-CoggC2Enmdbu_sptTMZFw3b-suM5YzzjVTmSLu3QByLPD0FAsfIAmlKQlNRV2RVh-D_Rmz_ygRpaC6bLeyJSyygearE2qYPrYLIg_YRl_6nEK50cs/s640/three-single-page-bffs-behind-proxy.png" width="640" /></a></div>
<br />
This is a standard thing to do and can easily be done with widespread technology, like <a href="http://nginx.org/">nginx</a>, <a href="http://www.squid-cache.org/">squid</a> or IIS with <a href="https://www.iis.net/downloads/microsoft/application-request-routing"> ARR</a>. In other words: There is nothing new here, I am only pointing out that using the single-page BFF pattern will lead a need to set a up a reverse proxy in front of your services.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-89753706667529756932016-09-07T21:16:00.001+02:002016-09-07T21:16:47.398+02:00Single-Page BFFsIn this post I describe a variation of the backend for frontend (AKA BFF) pattern that <a href="https://twitter.com/strudso">Søren Trudsø</a> made me aware of. With this variation we create not just a backend for each frontend - one backend for the iOS app, one for the Android app and one for the web site, say - but a backend for each page in the web frontend: A single-page BFF.<br />
<br />
The BFF patterns is a pattern for building applications on top of a system of microservices. I describe that pattern in my <a href="https://manning.com/books/microservices-in-net-core?a_aid=horsdalconsult&a_bid=66b54382">Microservice in .NET Core</a> book, but you can also find good descriptions of the pattern on the web - for instance <a href="http://samnewman.io/patterns/architectural/bff/">from Sam Newman</a>. To sum it up the BFF patterns is that you create a microservice for each frontend you have. That is, if you have an iOS app there is an iOS BFF. If you have an Android app you have an Android BFF. If you have an Oculus Rift app there is an Oculus Rift BFF. Each BFF is a microservice with the sole responsibility of serving its frontend - the Android BFF is there solely to serve the Android app and does not care about the Oculus rift app whatsoever. The BFFs do nothing more than gather whatever their frontends - or apps - need and serve that in a format convenient to the frontend. The BFFs do not implement any business logic themselves, all that is delegated to other microservices. This figure illustrates the setup:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieCrXMD7Gs43hkAGBifAlTcdCnFMf7Xn93TWWDurlkSdONqvmgVBo93x0aO0SFBy88SNa-joQSS6GqiThLZn618Y4hDRVRpgyxFj8MAy5ZlToGvvlHApvSkbnaUF195y85xB3qSkLKyMw/s1600/bffs.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieCrXMD7Gs43hkAGBifAlTcdCnFMf7Xn93TWWDurlkSdONqvmgVBo93x0aO0SFBy88SNa-joQSS6GqiThLZn618Y4hDRVRpgyxFj8MAy5ZlToGvvlHApvSkbnaUF195y85xB3qSkLKyMw/s640/bffs.jpg" width="640" /></a></div>
<br />
In this setup each BFF tends to grow as its frontend grows. That is, the web BFF tends to grow as the web frontend grows: When more functionality is added to existing pages, that functionality might need new endpoints to make AJAX requests to, and thus the web BFF grows a little bit. When new pages are added to the web frontend, the web BFF also grows.<br />
<br />
<i>Sidenote: I realize that the term "page" on a web site is somewhat fuzzy these days: Single page apps routinely swap the entire view from one thing to something completely different, giving the user the experience of going to a new "page". In this post I use the term "page" in the more traditional sense of a full page reload. You know, that thing when you follow a link and the browser loads a completely new HTML document from a new URL. I think you've encountered it before :D</i><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtXFWR_ioTh7QQKHBlIqVFs5EJjy-TgGLRQFbiH0oQxe6NDJT72SSp5lVs-pq74haweSVIZOOxu1OWVo3HH_sDI8jIkYjTvNr3_DvBGeQ1ebcbMFkUI_QtRgSTvfvIT06r7-673-HlVZE/s1600/web-bff-grows.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="338" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtXFWR_ioTh7QQKHBlIqVFs5EJjy-TgGLRQFbiH0oQxe6NDJT72SSp5lVs-pq74haweSVIZOOxu1OWVo3HH_sDI8jIkYjTvNr3_DvBGeQ1ebcbMFkUI_QtRgSTvfvIT06r7-673-HlVZE/s640/web-bff-grows.jpg" width="640" /></a></div>
<br />
The size of the web BFF might not be problem at first (or ever), but a some point enough may have been added to the web frontend to make it a problem. In this situation I have found it useful to break the web BFF down by page boundaries: In stead of having one BFF serve the entire web frontend, I will have a one BFF for each page on the web site, like so:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3sKxQ-zcpTMxjbC6ztubisFldbvHxGKH4j8yHgg-eHWbRL1Oq3tGPNLgCsWq4STbem0M9JuP0RTn1ONCcwO-ym1ss_0nMka3Qh4J2x28CR4a2FqrNn7OVGg6xi_2f96sMt5aXzTn0Q_A/s1600/single-page-bff.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="339" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3sKxQ-zcpTMxjbC6ztubisFldbvHxGKH4j8yHgg-eHWbRL1Oq3tGPNLgCsWq4STbem0M9JuP0RTn1ONCcwO-ym1ss_0nMka3Qh4J2x28CR4a2FqrNn7OVGg6xi_2f96sMt5aXzTn0Q_A/s640/single-page-bff.jpg" width="640" /></a></div>
<br />
This way the BFFs are kept small and focused on a single task, namely serving a single page.<br />
<br />
Notice that one or more of the pages here can be single page apps that include several views, so there need not be a direct correspondance between what the use perceives a separate views - or pages - and the single page BFFs on the backend. Rather, in such cases, there is a BFF for each single page app.Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0tag:blogger.com,1999:blog-8336318365940619669.post-84062835422112009222016-02-17T13:08:00.000+01:002016-02-20T21:40:36.271+01:00Book Excerpt: Expecting Failures In Microservices and Working Around Them<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0IRgtOQ7cvzs9NBsRiWv2_eeAswbPhahxW5MgUGzzqr7YxG0N_bUnhMQYz_dbWPrEjYA9n7Msbvm7Lf0ZqI3X077AoCRvT0w0b-lEm46YFYf4e8Z3J1jYaNnpW9Bn18zbHpgMtwgYrnY/s1600/book-cover.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0IRgtOQ7cvzs9NBsRiWv2_eeAswbPhahxW5MgUGzzqr7YxG0N_bUnhMQYz_dbWPrEjYA9n7Msbvm7Lf0ZqI3X077AoCRvT0w0b-lEm46YFYf4e8Z3J1jYaNnpW9Bn18zbHpgMtwgYrnY/s200/book-cover.jpg" width="159" /></a><br />
This article was excerpted from the book <a href="https://www.manning.com/books/microservices-in-net">Microservices in .NET</a>.<br />
<br />
When working with any non-trivial software systems,<span lang="X-NONE"> we must expect failures to occur. Hardware can fail. The software itself might fail due, for instance, to unforeseen usage or corrupt data. A distinguishing factor of a microservice system is that there is a lot of communication between the microservices. <br />
<br />
Figure 1 shows the communication resulting from a user adding an item to his/her shopping cart. From figure 1 we see that just one user action results in a good deal of communication. Considering that a system will likely have concurrent users all performing many actions, we can see that there really is a lot of communication going on inside a microservice system. <br />
<br />
We must expect that communication to fail from time to time. The communication between only two microservices may not fail very often, but in regard to a microservice system as a whole, communication failures are likely to occur often simply because of the amount of communication going on</span><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixZXp5L3sLnBJ_rEemkySde2bdruj47yKpExwtXN7urVCLD_glgY9AEP_PzygiaDCZq_J8vnCXLWad9tolQT_xO5unvf1rPTRKwkYpZhs4oYczbYlcAvwXBYGNlcdjoSmgH31e9-GT5Mo/s1600/interaction-overview.jpg" imageanchor="1"><img border="0" height="408" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixZXp5L3sLnBJ_rEemkySde2bdruj47yKpExwtXN7urVCLD_glgY9AEP_PzygiaDCZq_J8vnCXLWad9tolQT_xO5unvf1rPTRKwkYpZhs4oYczbYlcAvwXBYGNlcdjoSmgH31e9-GT5Mo/s640/interaction-overview.jpg" width="640" /></a></div><div class="Body" style="text-align: center;"><span style="font-family: "times new roman" , serif; font-size: 10pt;">Figure 1 In a system of microservices, there will be many communication paths</span></div><div class="Body"><br />
Since we have to expect that some of the communication in our microservice system will fail, we should design our microservices to be able to cope with those failures.<br />
<br />
We can divide the collaborations between microservices into three categories: Query, command and event based collaborations. When a communication fails, the impact depends on the type of collaboration and way the microservices cope with it:<o:p></o:p></div><div class="ListBulletCxSpFirst"></div><ul><li><span style="font-family: "symbol";"><span style="font-family: "times new roman"; font-size: 7pt; font-stretch: normal;"> </span></span><i>Query</i> based collaboration: When a query fails, the caller does not get the information it needs. If the caller copes well with that, the impact is that the system keeps on working, but with some degraded functionality. If the caller does not cope well, the result could be an error.</li>
<li><i>Command</i> based collaboration: When sending a command fails, the sender won’t know if the receiver got the command or not. Again, depending on how the sender copes, this could result in an error, or it could result in degraded functionality.</li>
<li><i>Event</i> based collaboration: When a subscriber polls an event feed, but the call fails, the impact is limited. The subscriber will poll the event feed later and, assuming the event feed is up again, receive the events at that time. In other words, the subscriber will still get all events, only some of them will be delayed. This should not be a problem for an event-based collaboration, since it is asynchronous anyway.</li>
</ul><br />
<div class="ListBulletCxSpLast"><o:p></o:p></div><div class="Head2"><span lang="X-NONE"><b><span style="font-size: large;">Have Good Logs</span></b><o:p></o:p></span></div><br />
<div class="Body1"><span lang="X-NONE">Once we accept that failures are bound to happen and that some of them may result, not just in a degraded end user experience, but in errors, we must make sure that we are able to understand what went wrong when an error occurs. That means that we need good logs that allow us to trace what happened in the system leading up to an error situation. "What happened" will often span several microservices, which is why you should consider introducing a central <i>Log Microservice</i>, as shown in figure 2, that all the other microservices send log messages to, and which allows you to inspect and search the logs when you need to.<o:p></o:p></span></div><div class="Body1"><span lang="X-NONE"><br />
</span></div><div class="Body1"><span lang="X-NONE"><br />
</span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_x5wqGG23eHDqJ1wG1ggwpC5BRgb2alJm5bwWM6ABrzi0FVkNzQiqPlg7G-hxieccjITQGI96mgyiqDO-y26eWB3tlL0SXw7Gc3kAkgLdOK5a8bhnY1QY58tqKjxqzoPzr_f-et8ngUI/s1600/overview-with-logging-service.jpg" imageanchor="1"><img border="0" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_x5wqGG23eHDqJ1wG1ggwpC5BRgb2alJm5bwWM6ABrzi0FVkNzQiqPlg7G-hxieccjITQGI96mgyiqDO-y26eWB3tlL0SXw7Gc3kAkgLdOK5a8bhnY1QY58tqKjxqzoPzr_f-et8ngUI/s640/overview-with-logging-service.jpg" width="640" /></a></div><div class="FigureBCaption" style="text-align: center;"><span style="font-size: x-small;">Figure 2 A central Log Microservice receives log messages from all other microservices and stores them in a database or a search engine. The log data is accessible through a web interface. The dotted arrows show microservices sending log messages to the central Log Microservice<o:p></o:p></span></div><div class="Body"><br />
</div><div class="Body">The <i>Log Microservice</i> is a central component that all other microservices use. We need to make certain that a failure in the <i>Log Microservice</i> does not bring down the whole system when all other microservices fail because they are not able to log messages. Therefore, sending log messages to the <i>Log Microservice</i> must be <i>fire and forget</i> - that is, the messages are sent and then forgotten about. The microservice sending the message should not wait for a response.</div><div class="Body"><o:p></o:p></div><div class="Body"><br />
</div><table border="0" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="border-collapse: collapse; margin-left: -4.5pt; mso-padding-alt: 5.4pt 2.3pt 0in 0in; mso-yfti-tbllook: 1184; width: 501px;"><tbody>
<tr style="height: 138.3pt; mso-yfti-firstrow: yes; mso-yfti-irow: 0; mso-yfti-lastrow: yes;"> <td style="background: rgb(204, 204, 204); height: 138.3pt; padding: 5.4pt 2.3pt 0in 0in; width: 58.5pt;" valign="top" width="78"><div align="left" class="MsoNormal" style="line-height: 106%; margin: 6pt 0in 0.0001pt 2.5pt;"><span style="color: #444444;"><b><span style="font-family: "arial" , sans-serif; font-size: 9pt; line-height: 106%;">SIDEBAR</span></b><span style="font-family: "calibri" , sans-serif; font-size: 9pt; line-height: 106%;"><o:p></o:p></span></span></div></td> <td style="background: rgb(204, 204, 204); height: 138.3pt; padding: 5.4pt 2.3pt 0in 0in; width: 316.95pt;" valign="top" width="423"><div align="left" class="MsoNormal" style="line-height: 106%; margin-bottom: 1.1pt;"><span style="color: #444444;"><b><span style="font-family: "arial" , sans-serif; font-size: 9.0pt; line-height: 106%;">Use an Off-the-Shelf Solution for the Log Microservice</span></b><span style="font-family: "calibri" , sans-serif; font-size: 9.0pt; line-height: 106%;"><o:p></o:p></span></span></div><div class="MsoNormal" style="line-height: 106%; margin-bottom: .0001pt; margin-bottom: 0in; margin-left: 0in; margin-right: 33.7pt; margin-top: 6.0pt;"><span style="color: #444444;"><span><span style="font-family: "arial" , sans-serif; font-size: 9pt; line-height: 106%;">A central <i>Log Microservice</i> does not implement a business capability of a particular system. It is an implementation of generic technical capability. In other words the requirements to a <i>Log Microservice</i> in systems A are not that different from the requirements to a <i>Log Microservice</i> is system B. Therefore I recommend using an off-the-shelf solution to implement your <i>Log Microservice</i> - for instance logs can be stored in </span><span style="font-family: "calibri" , sans-serif; font-size: 9pt; line-height: 106%;"><a href="https://github.com/elastic/elasticsearch"><span style="font-family: "arial" , sans-serif; mso-fareast-font-family: Arial;">Elasticsearch </span></a></span><span style="font-family: "arial" , sans-serif; font-size: 9pt; line-height: 106%;">and made accessible with </span><span style="font-family: "calibri" , sans-serif; font-size: 9pt; line-height: 106%;"><a href="https://github.com/elastic/kibana"><span style="font-family: "arial" , sans-serif; mso-fareast-font-family: Arial;">Kibana</span></a></span><span style="font-family: "arial" , sans-serif; font-size: 9pt; line-height: 106%;">. These are well-established and well-documented products, but I will not delve into how to set them up here. </span></span><span style="font-family: "calibri" , sans-serif; font-size: 9pt; line-height: 106%;"><o:p></o:p></span></span></div></td> </tr>
</tbody></table><h4><span style="font-weight: normal;"><br />
</span></h4><h3><span style="font-weight: normal;">Correlation Tokens</span></h3><div class="Body1"><span lang="X-NONE">In order to be able to find all log messages related to </span>a <span lang="X-NONE">particular action in the system</span>,<span lang="X-NONE"> we can use <i>correlation tokens</i>. A correlation token is an identifier attached to a request from an end user when it comes into the system. The correlation token is passed along from microservice to microservice in any communication that stems from that end-user request. Any time one of the microservices sends a log message to the <i>Log Microservice</i></span><i>, </i><span lang="X-NONE">the message should include the correlation token. The <i>Log Microservice</i> should allow searching for log messages by correlation token. Referring to figure 2, the <i>API Gateway </i>would create and assign a correlation token to each incoming request. The correlation is then passed along with every microservice-to-microservice communication.<o:p></o:p></span></div><div class="Head2"><span lang="X-NONE"><br />
</span></div><div class="Head2"><span lang="X-NONE"><b><br />
</b></span></div><div class="Head2"><span style="font-size: large;"><span lang="X-NONE"><b>Roll forward vs Roll back</b></span><span lang="X-NONE"><o:p></o:p></span></span></div><div class="Head2"><span lang="X-NONE"><b><br />
</b></span></div><div class="Body1"><span lang="X-NONE">When errors happen in production, we are faced with the question of how to fix </span>them<span lang="X-NONE">. In many traditional systems, if errors start occurring shortly after a deployment, the default would be to roll back to the previous version of the system. In a microservice system, the default can be different. </span>M<span lang="X-NONE">icroservices lend themselves to continuous delivery. With continuous delivery, microservices will be deployed very often and each deployment should be both fast and easy to perform. Furthermore, microservices are sufficiently small and simple </span>so<span lang="X-NONE"> many bug fixes are also simple. This opens the possibility of <i>rolling forward</i> rather than rolling backward.<o:p></o:p></span></div><div class="Body"><br />
</div><div class="Body">Why would we want to default to rolling forward instead of rolling backward? In some situations, rolling backward is complicated, particularly when database changes are involved. When a new version that changes the database is deployed, the microservice will start producing data that fits in the updated database. Once that data is in the database, it has to stay there, which may not be compatible with rolling back to an earlier version. In such a case, rolling forward might be easier.<o:p></o:p></div><div class="Head2"><span lang="X-NONE"><br />
</span></div><div class="Head2"><span lang="X-NONE"><b><br />
</b></span></div><div class="Head2"><span lang="X-NONE"><b><span style="font-size: large;">Do Not Propagate Failures</span></b><o:p></o:p></span></div><div class="Head2"><span lang="X-NONE"><b><br />
</b></span></div><div class="Body1"><span lang="X-NONE">Sometimes things happen around a microservice that may disturb the normal operation of the microservice. We say that the microservice i</span>s<span lang="X-NONE"> under stress in such situations. There are many sources of stress. To name a few, a microservice may be under stress because</span>:</div><div class="Body1"></div><ul><li><span style="font-family: "symbol";"><span style="font-family: "times new roman"; font-size: 7pt; font-stretch: normal;"> </span></span>One of the machines in the cluster its data store runs on has crashed</li>
<li><span style="font-family: "symbol";"><span style="font-family: "times new roman"; font-size: 7pt; font-stretch: normal;"> </span></span>It has lots network connectivity to one of its collaborators</li>
<li>It is receiving unusually high amounts of traffic</li>
<li>One of its collaborators is down</li>
</ul><br />
<div class="ListBulletCxSpLast"><o:p></o:p></div><div class="Body">In all of these situations, the microservice under stress cannot continue to operate the way it normally does. That doesn’t mean that it’s down, only that it must cope with the situation.<o:p></o:p></div><div class="Body"><br />
</div><div class="Body">When one microservice fails, its collaborators are put under stress and are also at risk of failing. While the microservice is failing, its collaborators will not be able to query, send commands or poll events from the failing microservice. As illustrated in figure 3, if this makes the collaborators fail, even more microservices are at risk of failing. At this point, the failure has started propagating through the system of microservices. Such a situation can quickly escalate from one microservice failing to lot of microservices failing.<o:p></o:p></div><div class="Body1"><span lang="X-NONE"> <span style="font-family: "times new roman" , serif; font-size: 10pt;"><br clear="all" style="mso-special-character: line-break; page-break-before: always;" /></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM08R19qSMQub3hfsm6b7JHEPhDrBr1pLKBC91177QyRgqxm1aJRV9XyBY1HpNE5afojbuIlPl1gYWX9vW5ErcA_0xz66lF0bRI7RXT9hpZV09fO0I6kL7Er9bHRugab21gl7kfh9Ud1c/s1600/failure-propagatio.jpg" imageanchor="1"><img border="0" height="488" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM08R19qSMQub3hfsm6b7JHEPhDrBr1pLKBC91177QyRgqxm1aJRV9XyBY1HpNE5afojbuIlPl1gYWX9vW5ErcA_0xz66lF0bRI7RXT9hpZV09fO0I6kL7Er9bHRugab21gl7kfh9Ud1c/s640/failure-propagatio.jpg" width="640" /></a></div><div class="Body1"><span lang="X-NONE"><span style="font-family: "times new roman" , serif; font-size: 10pt;"><br />
</span></span></div><div class="FigureBCaption" style="text-align: center;"><span style="font-size: x-small;">Figure 3 If the microservice marked FAILED is failing, so is the communication with it. That means that the microservices at the other end of those communications are under stress. If the stressed microservices fail due to the stress, the microservices communicating with them are put under stress. In that situation, the failure in the failed microservice has propagated to several other microservices.</span><o:p></o:p></div><div class="Body"><br />
</div><div class="Body">Some examples of how we can stop failures propagating are:<o:p></o:p></div><div class="ListBulletCxSpFirst"></div><ul><li>When one microservice tries to send a command to another microservice, which happens to be failing at the time, that request will fail. If the sender simply fails as well, we get the situation illustrated in figure 3 where the failures propagate back through the system. To stop the propagation, the sender might act as if the command succeeded, but actually store the command into a list of failed commands. The sending microservice can periodically go through the list of failed commands and try to send them again. This is not possible in all situations, because the command may need to be handled here and now, but when this approach is possible it stops the failure in one microservice from propagating.</li>
<li>When one microservice queries another one that’s failing, the caller could use a cached response. In case the caller has a stale response in the cache, but a query for a fresh response fails, it might decide to use the stale response anyway. Again, this is not something that will be possible in all situations, but when it is, the failure will not propagate.</li>
<li>An <i>API Gateway</i> that is stressed because of high amounts of traffic from a certain client can throttle that client by not responding to more than a certain number of requests per second from that client. Notice that the client may be sending an unusually high amount of requests because it is somehow failing internally. When throttled, the client will get a degraded experience, but will still get some responses. Without the throttling, the <i>API Gateway</i> might become slow for all clients or it might fail completely. Moreover, since the <i>API Gateway</i> collaborates with other microservices, handling all the incoming requests would push the stress of those requests onto other microservices too. Again, the throttling stops the failure in the client from propagating further into the system to other microservices.</li>
</ul><br />
<div class="ListBulletCxSpLast"><o:p></o:p></div><div class="Body">As we can see from these examples, stopping failure propagation comes in many shapes and sizes. The important thing to take away from this article is the idea of building safeguards into your systems that are specifically designed to stop from propagating the kinds of failures you anticipate. How that is realized depends on the specifics of the systems you are building. Building in these safeguards may take some effort, but it’s very often well worth the effort because of the robustness they give the system as a whole.<o:p></o:p></div><div class="Body1"><span lang="X-NONE"> </span></div><div class="Body1"><br />
</div>Christian Horsdalhttp://www.blogger.com/profile/07782376855026802670noreply@blogger.com0