From d5de65fdb1802cdf498d65d93397f290813c377c Mon Sep 17 00:00:00 2001
From: muqiuhan <muqiuhan@users.noreply.github.com>
Date: Tue, 9 Sep 2025 06:17:00 +0000
Subject: deploy: 9665097f0fa0dae9f92123fac54d26c0818758a5

---
 .../index.html                                     | 95 ++++++++++------------
 1 file changed, 41 insertions(+), 54 deletions(-)

(limited to '2024/10/18/Building-custom-fibers-library-in-FSharp/index.html')

diff --git a/2024/10/18/Building-custom-fibers-library-in-FSharp/index.html b/2024/10/18/Building-custom-fibers-library-in-FSharp/index.html
index 73182fbe..42f0d0eb 100644
--- a/2024/10/18/Building-custom-fibers-library-in-FSharp/index.html
+++ b/2024/10/18/Building-custom-fibers-library-in-FSharp/index.html
@@ -202,7 +202,8 @@
 <p>The general idea is simple - we want a fine-grained concurrency primitive, that will let us easily compose chain of operations in sequential manner. Of course we could use threads here, but the question is: are threads fine-grained? In many managed languages with OS threads exposed, they can be quite heavy eg. by default in .NET each thread takes around 1MB of memory and requires calling kernel code to cooperate with other threads, which is an expensive operation on its own.</p>
 <p>What we’re after, are more lightweight structures (less than 1kB), that can live fully in a user space, so that we can have even millions of them cooperating frequently with each other without heavy performance penalties.</p>
 <p>Before we begin, I think it’s good to discuss different designs. We’ll cover several different topics to be able to make more informed decisions, that we’re up to apply to our own solution.</p>
-<h3 id="Preemptive-vs-cooperative-scheduler"><a href="#Preemptive-vs-cooperative-scheduler" class="headerlink" title="Preemptive vs cooperative scheduler"></a>Preemptive vs cooperative scheduler</h3><p>Scheduler is a subsystem, which direct responsibility is to assign CPU core processing power to a particular fiber. It’s also responsible for coordinating fibers execution. The two most common categories of schedulers are preemptive and cooperative.</p>
+<h3 id="Preemptive-vs-cooperative-scheduler"><a class="header-anchor" href="#Preemptive-vs-cooperative-scheduler">¶</a>Preemptive vs cooperative scheduler</h3>
+<p>Scheduler is a subsystem, which direct responsibility is to assign CPU core processing power to a particular fiber. It’s also responsible for coordinating fibers execution. The two most common categories of schedulers are preemptive and cooperative.</p>
 <p>A preemptive scheduler is the one, that’s always in control of fiber execution. It’s able to decide on its own, when fiber can be started and stopped. The most obvious example of such is a thread scheduler existing on most operating systems.</p>
 <p>Preemptive scheduler usually works in one of two ways:</p>
 <ul>
@@ -213,20 +214,24 @@
 <p>A cooperative scheduler doesn’t have a concept of preemption - once started by the scheduler, a fiber will execute until it doesn’t give back the control willingly. This is often done with dedicated programming constructs, and often is known as yielding, parking or awaiting.</p>
 <p>In cooperative variant, a fiber body is usually split into series of discrete steps, between which fiber gives control back to the scheduler.</p>
 <p>Keep in mind that these two are not mutually exclusive - a preemptive scheduler often provides a way for a fiber to return control back to it when it’s known that fiber won’t be executing any longer eg. because it has been put to sleep for a while.</p>
-<h3 id="Stackless-vs-stackful"><a href="#Stackless-vs-stackful" class="headerlink" title="Stackless vs. stackful"></a>Stackless vs. stackful</h3><p>A concept, that’s somewhat related to a topic above is the idea of stackless and stackful coroutines.</p>
-<p>A stackful variant is aware of underlying execution stack and can preserve&#x2F;restore parts of it when yielding&#x2F;continuing a fiber. Examples of this approach could be Go, Lua, Python asyncio and in the future, also <a target="_blank" rel="noopener" href="https://www.youtube.com/watch?v=NV46KFV1m-4&ref=bartoszsypytkowski.com">Java Loom</a> project. Implementing such option (if it’s not implemented by a runtime already) usually requires diving deep into low-level internals, since execution stack is not something that most managed languages offers the users to play with, and doing so without coordination with runtime can cause problems - like determining liveness of objects for GC purposes.</p>
+<h3 id="Stackless-vs-stackful"><a class="header-anchor" href="#Stackless-vs-stackful">¶</a>Stackless vs. stackful</h3>
+<p>A concept, that’s somewhat related to a topic above is the idea of stackless and stackful coroutines.</p>
+<p>A stackful variant is aware of underlying execution stack and can preserve/restore parts of it when yielding/continuing a fiber. Examples of this approach could be Go, Lua, Python asyncio and in the future, also <a target="_blank" rel="noopener" href="https://www.youtube.com/watch?v=NV46KFV1m-4&amp;ref=bartoszsypytkowski.com">Java Loom</a> project. Implementing such option (if it’s not implemented by a runtime already) usually requires diving deep into low-level internals, since execution stack is not something that most managed languages offers the users to play with, and doing so without coordination with runtime can cause problems - like determining liveness of objects for GC purposes.</p>
 <p>Stackless coroutine usually captures locals that we want to preserve as part of callback object (lambda), that is allocated as an object on the heap and scheduled on yield continuation. These steps are usually visible directly in code (eg. await in C#, Rust and JavaScript, but also joints of Scala for-comprehensions, bang-suffix in F# or Haskell do-notation), but sometimes can be implicit like in case of Kotlin. Take into account that while many languages offer syntax support for those constructs, it’s not explicitly necessary to work - take a look at JavaScript and <code>Promise.then</code> as an example.</p>
 <p>Stackless coroutines usually construct their logic around one of two concepts:</p>
 <ol>
 <li>Finite state machines - this variant is usually faster and can be encoded manually (example of such case is Akka actors), but for a human eye it usually doesn’t really read as a sequential step-by-step program execution, unless it has some support from the compiler itself (see: C# and Rust).</li>
-<li>Monadic sequencing via bind&#x2F;flatMap operator, which is very popular in functional languages. While we cover it in more details in the rest of this blog post, for now it’s enough to say that it’s a way to chain callback-based behaviors together in a way, that resembles standard sequential code.</li>
+<li>Monadic sequencing via bind/flatMap operator, which is very popular in functional languages. While we cover it in more details in the rest of this blog post, for now it’s enough to say that it’s a way to chain callback-based behaviors together in a way, that resembles standard sequential code.</li>
 </ol>
-<p>For sure one of the advantages of stackful coroutines is that they’re <em>mono-colored</em>: you can yield&#x2F;continue coroutine execution from within any other function, while in the stackless variant splits your world into <em>two-colored</em> functions - synchronous and asynchronous - where async one can be only called and yielded safely (without blocking underlying OS thread) from within another async function.</p>
-<h3 id="Eager-vs-lazy-fibers"><a href="#Eager-vs-lazy-fibers" class="headerlink" title="Eager vs lazy fibers"></a>Eager vs lazy fibers</h3><p>We already mentioned two important events in fiber execution life cycle - starting and parking. Here I briefly discuss about different design decisions on when to start a fiber execution.</p>
+<p>For sure one of the advantages of stackful coroutines is that they’re <em>mono-colored</em>: you can yield/continue coroutine execution from within any other function, while in the stackless variant splits your world into <em>two-colored</em> functions - synchronous and asynchronous - where async one can be only called and yielded safely (without blocking underlying OS thread) from within another async function.</p>
+<h3 id="Eager-vs-lazy-fibers"><a class="header-anchor" href="#Eager-vs-lazy-fibers">¶</a>Eager vs lazy fibers</h3>
+<p>We already mentioned two important events in fiber execution life cycle - starting and parking. Here I briefly discuss about different design decisions on when to start a fiber execution.</p>
 <p>Eager execution means, that fiber is started automatically after its creation. An example of such are Scala <code>Future[A]</code> and JavaScript <code>Promise</code>. Since execution process starts right away, we’re willingly resign from a certain degree of control over how or when to execute given fiber. Usually this is solved by wrapping a fiber creation into another function or lambda.</p>
 <p>Lazy execution is much more common and preferred way of work, as it allows us to separate place where we want to define our asynchronous sequence of steps from the place, where the execution details are defined. It’s used in C# TPL as well as pretty much in all functional languages implementations (excluding Scala futures mentioned earlier).</p>
-<h3 id="Interruption"><a href="#Interruption" class="headerlink" title="Interruption"></a>Interruption</h3><p>There are also few decisions regarding premature escaping the fiber execution, also known as interruption&#x2F;cancelation: one of them requires passing special object - a token - between method calls and explicit checking for its completion. It is how C# Tasks work. However putting such requirement onto the API user can be cumbersome and error-prone option. Therefore pretty much every other coroutine library either allows to direcly interrupt a fiber or (like in case of F# Async) passes cancelation tokens and check if they were triggered under the hood.</p>
-<h2 id="Implementation"><a href="#Implementation" class="headerlink" title="Implementation"></a>Implementation</h2><p>Since we talked a bit about various approaches, let’s get to the meat of this blog post: implementing our own coroutine library in F#. So, what properties will it have?:</p>
+<h3 id="Interruption"><a class="header-anchor" href="#Interruption">¶</a>Interruption</h3>
+<p>There are also few decisions regarding premature escaping the fiber execution, also known as interruption/cancelation: one of them requires passing special object - a token - between method calls and explicit checking for its completion. It is how C# Tasks work. However putting such requirement onto the API user can be cumbersome and error-prone option. Therefore pretty much every other coroutine library either allows to direcly interrupt a fiber or (like in case of F# Async) passes cancelation tokens and check if they were triggered under the hood.</p>
+<h2 id="Implementation"><a class="header-anchor" href="#Implementation">¶</a>Implementation</h2>
+<p>Since we talked a bit about various approaches, let’s get to the meat of this blog post: implementing our own coroutine library in F#. So, what properties will it have?:</p>
 <ol>
 <li>We use cooperative scheduling (we don’t want to tweak the compiler) of stackless fibers with support from F# computation expression for nice syntax.</li>
 <li>We use simple approach by defining custom <code>bind</code> operator with support from F# computation expressions. No state machines.</li>
@@ -236,31 +241,26 @@
 <p>All of these give us in very similar approach to that found inside of native F# <code>Async</code> data type. To begin with, we’ll simply define the shape of our fiber.</p>
 <p>Underneath, pretty much every cooperative stackless coroutine approach uses callbacks to drive the flow of synchronous segments of code to be executed one after another. So what we need is a callback which takes a result of previous coroutine and schedules in within some context of execution:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">Fiber</span><span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">=</span> Fiber <span class="keyword">of</span> (ExecutionContext <span class="operator">-&gt;</span> FiberCallback<span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">-&gt;</span> <span class="type">unit</span>)</span><br></pre></td></tr></table></figure>
-
 <p>Here we’ll represent Fiber as a simple single-case discriminated union. We could as well define other specialized cases, like:</p>
 <ul>
 <li>Situation when coroutine is executed immediately and doesn’t need to be awaited on: think about variant of <code>ValueTask</code> from C# Task Parallel Library.</li>
 <li>Case when coroutine fails - in that case we might want to store an artificial tracing context that would allow us to create nicely-formatted “stack traces”: since .NET Core 2.1, C# already provides similar solution however AFAIK it’s been solved differently.</li>
 </ul>
-<p>Ok, but what are <code>ExecutionContext</code> and <code>FiberCallback&lt;&#39;a&gt;</code>? Let’s start from callback. We can represent it as follows:</p>
+<p>Ok, but what are <code>ExecutionContext</code> and <code>FiberCallback&lt;'a&gt;</code>? Let’s start from callback. We can represent it as follows:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">FiberCallback</span><span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">=</span> FiberResult<span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">-&gt;</span> unit</span><br></pre></td></tr></table></figure>
-
-<p>It’s just a simple function, which takes result of previous fiber execution and handles it. What’s the <code>FiberResult&lt;&#39;a&gt;</code> then?</p>
-<p>Our fiber can complete successfully (returning a value) or fail with an exception. We’ll be conservative here and won’t go into more typed world of <a target="_blank" rel="noopener" href="http://degoes.net/articles/bifunctor-io?ref=bartoszsypytkowski.com">IO bifunctor</a>. We can easy define these possible outputs in F# using <code>Result&lt;&#39;a, exn&gt;</code>.</p>
+<p>It’s just a simple function, which takes result of previous fiber execution and handles it. What’s the <code>FiberResult&lt;'a&gt;</code> then?</p>
+<p>Our fiber can complete successfully (returning a value) or fail with an exception. We’ll be conservative here and won’t go into more typed world of <a target="_blank" rel="noopener" href="http://degoes.net/articles/bifunctor-io?ref=bartoszsypytkowski.com">IO bifunctor</a>. We can easy define these possible outputs in F# using <code>Result&lt;'a, exn&gt;</code>.</p>
 <p>Question is: is that exhaustive? Well… no. As we already mentioned, there’s a 3rd state, often overlooked or conflated with failure: a canceled fiber. A canceled fiber doesn’t produce any output - since it was canceled before completion. In F# we already know how to represent an absence of value - simply use an option. Therefore our ultimate Fiber result type could look like this:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">FiberResult</span><span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">=</span> Result<span class="operator">&lt;</span><span class="symbol">&#x27;a</span>, exn<span class="operator">&gt;</span> option</span><br></pre></td></tr></table></figure>
-
 <p>Now, the <code>ExecutionContext</code>. While it can be compound of many different capabilities throughout the system - even to serve as functional equivalent of dependency injection - here I’ll use it only for implicit passing of specific scheduler info and cancelation tokens from one fiber to another.</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">ExecutionContext</span> <span class="operator">=</span> IScheduler <span class="operator">*</span> Cancel</span><br></pre></td></tr></table></figure>
-
 <p><code>IScheduler</code> interface is used to abstract component responsible for running our fibers. At the moment all we need is an ability to schedule fiber execution:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">[&lt;Interface&gt;]</span></span><br><span class="line"><span class="keyword">type</span> <span class="title class_">IScheduler</span> <span class="operator">=</span> </span><br><span class="line">  <span class="keyword">abstract</span> Schedule<span class="operator">:</span> (<span class="type">unit</span> <span class="operator">-&gt;</span> <span class="type">unit</span>) <span class="operator">-&gt;</span> <span class="type">unit</span></span><br></pre></td></tr></table></figure>
-
 <p>While the name and signature imply multithreaded execution model, it doesn’t have to be the case. We can even implement scheduler which will simulate everything on a single core.</p>
 <p>For now, we can simply implement a scheduler API on top of our standard .NET thread pool:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">module</span> Scheduler</span><br><span class="line"></span><br><span class="line"><span class="keyword">open</span> System.Threading</span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> shared <span class="operator">=</span> </span><br><span class="line">  &#123; <span class="keyword">new</span> ISchedule <span class="keyword">with</span></span><br><span class="line">      <span class="keyword">member</span> __.Schedule fn <span class="operator">=</span> </span><br><span class="line">        ThreadPool.QueueUserWorkItem(WaitCallback (<span class="keyword">fun</span> _ <span class="operator">-&gt;</span> fn())) <span class="operator">|&gt;</span> <span class="built_in">ignore</span>  &#125;</span><br></pre></td></tr></table></figure>
-
-<h3 id="Cancellation"><a href="#Cancellation" class="headerlink" title="Cancellation"></a>Cancellation</h3><p>Now it’s a time for cancellation tokens. Of course we could just make use of a flag - conceptually working like native .NET <code>CancellationToken</code>. However given implicit cancellation, it may not be enough. Example:</p>
+<h3 id="Cancellation"><a class="header-anchor" href="#Cancellation">¶</a>Cancellation</h3>
+<p>Now it’s a time for cancellation tokens. Of course we could just make use of a flag - conceptually working like native .NET <code>CancellationToken</code>. However given implicit cancellation, it may not be enough. Example:</p>
 <blockquote>
 <p>Imagine, that inside our fiber we’re scheduling the race between two other fibers ie. one writing data to a file and other which will complete after timeout. Now, whenever one of them completes first, we want to cancel another one to stop wasting resources for result that no longer matters.</p>
 </blockquote>
@@ -272,21 +272,21 @@
 </ol>
 <p>This behavior implies at least using two separate tokens, however in practice it will be more pragmatic to make our <code>Cancel</code> token work as a tree hierarchy - this way we can easily keep track of things and support more complex scenarios.</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">[&lt;Sealed;AllowNullLiteral&gt;]</span></span><br><span class="line"><span class="keyword">type</span> <span class="title class_">Cancel</span>(parent<span class="operator">:</span> Cancel) <span class="operator">=</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> flag<span class="operator">:</span> <span class="type">int</span> <span class="operator">=</span> <span class="number">0</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> children<span class="operator">:</span> Cancel <span class="type">list</span> <span class="operator">=</span> []</span><br><span class="line">  <span class="keyword">new</span>() <span class="operator">=</span> Cancel(<span class="literal">null</span>)</span><br><span class="line">  <span class="comment">/// Check if token was cancelled</span></span><br><span class="line">  <span class="keyword">member</span> __.Cancelled <span class="operator">=</span> flag <span class="operator">=</span> <span class="number">1</span></span><br><span class="line">  <span class="comment">/// Remove child token</span></span><br><span class="line">  <span class="keyword">member</span> <span class="keyword">private</span> __.RemoveChild(child) <span class="operator">=</span> </span><br><span class="line">    <span class="keyword">let</span> <span class="keyword">rec</span> loop child <span class="operator">=</span></span><br><span class="line">      <span class="keyword">let</span> children&#x27; <span class="operator">=</span> children</span><br><span class="line">      <span class="keyword">let</span> nval <span class="operator">=</span> children&#x27; <span class="operator">|&gt;</span> List.filter ((<span class="operator">&lt;&gt;</span>) child)</span><br><span class="line">      <span class="keyword">if</span> <span class="built_in">not</span> (obj.ReferenceEquals(children&#x27;, Interlocked.CompareExchange(<span class="operator">&amp;</span>children, nval, children&#x27;)))</span><br><span class="line">      <span class="keyword">then</span> loop child</span><br><span class="line">    <span class="keyword">if</span> <span class="built_in">not</span> (List.isEmpty children) <span class="keyword">then</span> loop child</span><br><span class="line">  <span class="comment">/// Create a new child token and return it.</span></span><br><span class="line">  <span class="keyword">member</span> this.AddChild () <span class="operator">=</span></span><br><span class="line">    <span class="keyword">let</span> <span class="keyword">rec</span> loop child <span class="operator">=</span></span><br><span class="line">      <span class="keyword">let</span> children&#x27; <span class="operator">=</span> children</span><br><span class="line">      <span class="keyword">if</span> (obj.ReferenceEquals(children&#x27;, Interlocked.CompareExchange(<span class="operator">&amp;</span>children, child<span class="operator">::</span>children&#x27;, children&#x27;)))</span><br><span class="line">      <span class="keyword">then</span> child</span><br><span class="line">      <span class="keyword">else</span> loop child</span><br><span class="line">    loop (Cancel this)</span><br><span class="line">  <span class="comment">/// Cancel a token</span></span><br><span class="line">  <span class="keyword">member</span> this.Cancel() <span class="operator">=</span></span><br><span class="line">    <span class="keyword">if</span> Interlocked.Exchange(<span class="operator">&amp;</span>flag, <span class="number">1</span>) <span class="operator">=</span> <span class="number">0</span> <span class="keyword">then</span></span><br><span class="line">      <span class="keyword">for</span> child <span class="keyword">in</span> Interlocked.Exchange(<span class="operator">&amp;</span>children, []) <span class="keyword">do</span> child.Cancel()</span><br><span class="line">      <span class="keyword">if</span> <span class="built_in">not</span> (isNull parent) <span class="keyword">then</span> parent.RemoveChild(this)</span><br><span class="line"></span><br></pre></td></tr></table></figure>
-
 <p>The general idea is simple: every new cancellation token (except root) may have a parent and a list of children. Canceling parent means canceling its children as well. After cancellation, we need to unpin child from its parent (therefore need for <code>RemveChild</code> operation) to avoid memory leaks.</p>
-<h5 id="Lock-free-updates"><a href="#Lock-free-updates" class="headerlink" title="Lock-free updates"></a>Lock-free updates</h5><p>What might be confusing for some in the code above, are recursive loops inside of <code>AddChild</code>&#x2F;<code>RemoveChild</code> operations. This is a good place to introduce lock-free algorithms: we use atomic operations from <a target="_blank" rel="noopener" href="https://www.bartoszsypytkowski.com/building-custom-fibers-library-in-f/docs.microsoft.com/en-us/dotnet/api/system.threading.interlocked">Interlocked</a> class to make sure that we can replace field references within a single CPU instruction, therefore making such field update safe without synchronized access. This is also known as Compare-And-Swap semantics.</p>
-<p>This alone however is not enough, as <code>Interlocked.CompareExchange(&amp;field, new&#39;, old)</code> can only safely replace a single field with new value if it contained an old one. This means that you cannot safely add or remove element to the list. So what can we do?</p>
+<h5 id="Lock-free-updates"><a class="header-anchor" href="#Lock-free-updates">¶</a>Lock-free updates</h5>
+<p>What might be confusing for some in the code above, are recursive loops inside of <code>AddChild</code>/<code>RemoveChild</code> operations. This is a good place to introduce lock-free algorithms: we use atomic operations from <a target="_blank" rel="noopener" href="https://www.bartoszsypytkowski.com/building-custom-fibers-library-in-f/docs.microsoft.com/en-us/dotnet/api/system.threading.interlocked">Interlocked</a> class to make sure that we can replace field references within a single CPU instruction, therefore making such field update safe without synchronized access. This is also known as Compare-And-Swap semantics.</p>
+<p>This alone however is not enough, as <code>Interlocked.CompareExchange(&amp;field, new', old)</code> can only safely replace a single field with new value if it contained an old one. This means that you cannot safely add or remove element to the list. So what can we do?</p>
 <ol>
 <li>We’re taking a value from the field.</li>
 <li>Update that value.</li>
 <li>Conditionally put it back again. What if in the meantime the field was already replaced by another concurrently running thread? In that case <code>Interlocked.CompareExchange</code> will return field value other that the one we read in step 1. This is why we compare its result with the variable we expected.</li>
-<li>If the expectation fails, we’ll retry - hence a recursive loop. Eventually even in high contention scenarios we should be able to complete after few retries. Given cheap and idempotent update operation, this still will be way faster than trying to call kernel code to obtain mutex&#x2F;semaphore lock.</li>
+<li>If the expectation fails, we’ll retry - hence a recursive loop. Eventually even in high contention scenarios we should be able to complete after few retries. Given cheap and idempotent update operation, this still will be way faster than trying to call kernel code to obtain mutex/semaphore lock.</li>
 </ol>
 <p>While this may sound like something error prone - we can potentially add the same element multiple times - in practice it’s safe, because our collection here is an immutable data structure. Adding the same element multiple times without updating the reference will always produce the same result.</p>
-<h3 id="Back-on-track…"><a href="#Back-on-track…" class="headerlink" title="Back on track…"></a>Back on track…</h3><p>Now we have pretty much all core structures. We’re ready to start building our fiber operators. Starting from the basic ones - a successfully completed fiber and the failed one:</p>
+<h3 id="Back-on-track…"><a class="header-anchor" href="#Back-on-track…">¶</a>Back on track…</h3>
+<p>Now we have pretty much all core structures. We’re ready to start building our fiber operators. Starting from the basic ones - a successfully completed fiber and the failed one:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> success r <span class="operator">=</span> Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (_, c) next <span class="operator">-&gt;</span> </span><br><span class="line">  <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span> <span class="keyword">else</span> next (<span class="literal">Some</span> (<span class="literal">Ok</span> r))</span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> fail ex <span class="operator">=</span> Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (_, c) next <span class="operator">-&gt;</span> </span><br><span class="line">  <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span> <span class="keyword">else</span> next (<span class="literal">Some</span> (<span class="literal">Error</span> ex))</span><br></pre></td></tr></table></figure>
-
-<p>Here, we simply pass a result&#x2F;error to our Fiber callback:</p>
+<p>Here, we simply pass a result/error to our Fiber callback:</p>
 <ul>
 <li>Cancelled fiber call <code>next</code> callback with <code>None</code> - as fibers cancelled before completion produce output .</li>
 <li>Successful call results in passing <code>Some (Ok result)</code> to a callback…</li>
@@ -295,41 +295,36 @@
 <p>You’ll be able to see a cancellation check made here as preamble of pretty much every operator body, which we’ll define. While it may sound cumbersome remember: we do that so that users of our fibers won’t have to :)</p>
 <p>Next very important operation is result mapping - we want to map result of one fiber into something else, returning another (lazy) fiber:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> mapResult (fn<span class="operator">:</span> <span class="type">Result</span><span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span> <span class="operator">-&gt;</span> <span class="type">Result</span><span class="operator">&lt;</span><span class="symbol">&#x27;b</span><span class="operator">&gt;</span>) (Fiber call) <span class="operator">=</span> Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">  <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">  <span class="keyword">else</span> </span><br><span class="line">    <span class="keyword">try</span> </span><br><span class="line">      call (s, c) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span></span><br><span class="line">        <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">        <span class="keyword">else</span> next (Option.map fn result))</span><br><span class="line">    <span class="keyword">with</span> e <span class="operator">-&gt;</span> next (<span class="literal">Some</span> (<span class="literal">Error</span> e))</span><br></pre></td></tr></table></figure>
-
 <p>We can use this function to compose more traditionally-looking <code>map</code> function…</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> map (fn<span class="operator">:</span> <span class="symbol">&#x27;a</span> <span class="operator">-&gt;</span> <span class="symbol">&#x27;b</span>) fiber <span class="operator">=</span> mapResult (Result.map fn) fiber</span><br></pre></td></tr></table></figure>
-
-<p>… however <code>mapResult</code> is more powerful - you could easily imagine using to apply failure recovery (a.k.a <code>try</code>&#x2F;<code>catch</code> semantics) by simply mapping <code>Error exception</code> → <code>Ok recoveredValue</code>:</p>
+<p>… however <code>mapResult</code> is more powerful - you could easily imagine using to apply failure recovery (a.k.a <code>try</code>/<code>catch</code> semantics) by simply mapping <code>Error exception</code> → <code>Ok recoveredValue</code>:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> catch fn fiber <span class="operator">=</span> mapResult (<span class="keyword">function</span> <span class="literal">Error</span> e <span class="operator">-&gt;</span> fn e <span class="operator">|</span> ok <span class="operator">-&gt;</span> ok) fiber</span><br></pre></td></tr></table></figure>
-
 <p>Another must-have function is binding operator (also know as <code>flatMap</code> in other languages like Scala, or <code>Promise.then</code> in JavaScript). It gives us the ability to compose fibers together - we’ll also use it when we come up to building a computation expression for our fibers.</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> bind (fn<span class="operator">:</span> <span class="symbol">&#x27;a</span> <span class="operator">-&gt;</span> Fiber<span class="operator">&lt;</span><span class="symbol">&#x27;b</span><span class="operator">&gt;</span>) (Fiber call) <span class="operator">=</span> Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">   <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span> </span><br><span class="line">   <span class="keyword">else</span> </span><br><span class="line">     <span class="keyword">try</span></span><br><span class="line">       call (s, c) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span></span><br><span class="line">         <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">         <span class="keyword">else</span> <span class="keyword">match</span> result <span class="keyword">with</span></span><br><span class="line">              <span class="operator">|</span> <span class="literal">Some</span> (<span class="literal">Ok</span> r) <span class="operator">-&gt;</span></span><br><span class="line">                 <span class="keyword">let</span> (Fiber call2) <span class="operator">=</span> fn r</span><br><span class="line">                 call2 (s, c) next <span class="comment">// pass `next` callback over to next fiber</span></span><br><span class="line">              <span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span> next <span class="literal">None</span></span><br><span class="line">              <span class="operator">|</span> <span class="literal">Some</span> (<span class="literal">Error</span> e) <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Error</span> e)))</span><br><span class="line">     <span class="keyword">with</span> e <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Error</span> e))</span><br></pre></td></tr></table></figure>
-
 <p>It’s simple - we execute one fiber from within another, passing the <code>next</code> callback from outer function as an argument to inner one.</p>
-<h3 id="Fiber-computation-expressions"><a href="#Fiber-computation-expressions" class="headerlink" title="Fiber computation expressions"></a>Fiber computation expressions</h3><p>With these few functions we’re already prepared to build a basic computation expression, that will enable us programming with fibers in pleasant way:</p>
+<h3 id="Fiber-computation-expressions"><a class="header-anchor" href="#Fiber-computation-expressions">¶</a>Fiber computation expressions</h3>
+<p>With these few functions we’re already prepared to build a basic computation expression, that will enable us programming with fibers in pleasant way:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">[&lt;Struct&gt;]</span></span><br><span class="line"><span class="keyword">type</span> <span class="title class_">FiberBuilder</span> <span class="operator">=</span></span><br><span class="line">    <span class="keyword">member</span> <span class="keyword">inline</span> __.Zero <span class="operator">=</span> Fiber.success (Unchecked.defaultof<span class="operator">&lt;</span>_<span class="operator">&gt;</span>)</span><br><span class="line">    <span class="keyword">member</span> <span class="keyword">inline</span> __.ReturnFrom fib <span class="operator">=</span> fib</span><br><span class="line">    <span class="keyword">member</span> <span class="keyword">inline</span> __.Return value <span class="operator">=</span> Fiber.success value</span><br><span class="line">    <span class="keyword">member</span> <span class="keyword">inline</span> __.Bind(fib, fn) <span class="operator">=</span> Fiber.bind fn fib</span><br><span class="line"></span><br><span class="line"><span class="meta">[&lt;AutoOpen&gt;]</span></span><br><span class="line"><span class="keyword">module</span> FiberBuilder <span class="operator">=</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">let</span> fib <span class="operator">=</span> FiberBuilder()</span><br></pre></td></tr></table></figure>
-
 <p>While in F# <a target="_blank" rel="noopener" href="https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/computation-expressions?ref=bartoszsypytkowski.com">there are many more operators</a> we could pack into our computation expression, these are basic ones that will let it work. With such construct, we’ll be able to write programs like:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="keyword">inline</span> millis n <span class="operator">=</span> TimeSpan.FromMilliseconds (float n)</span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> program<span class="operator">:</span> Fiber<span class="operator">&lt;</span><span class="type">int</span><span class="operator">&gt;</span> <span class="operator">=</span> <span class="keyword">fib</span> &#123;</span><br><span class="line">  <span class="keyword">let</span> a <span class="operator">=</span> <span class="keyword">fib</span> &#123;</span><br><span class="line">    <span class="keyword">do!</span> Fiber.delay (millis <span class="number">1000</span>) <span class="comment">// create some artificial delay</span></span><br><span class="line">    <span class="keyword">return</span> <span class="number">3</span></span><br><span class="line">  &#125;</span><br><span class="line">  <span class="keyword">let!</span> b <span class="operator">=</span> a <span class="operator">|&gt;</span> Fiber.timeout (millis <span class="number">3000</span>) <span class="comment">// execute task within specified timeout</span></span><br><span class="line">  <span class="keyword">return</span> b &#125;</span><br></pre></td></tr></table></figure>
-
 <p>Sure, we have neither delay nor timeout operators at the moment, but at least you know where are we heading now :)</p>
-<h3 id="Delayed-execution"><a href="#Delayed-execution" class="headerlink" title="Delayed execution"></a>Delayed execution</h3><p>In order to implement delays, we could theoretically just call <code>Thread.Sleep</code> and get over it, but this approach is devastating from any coroutine library point of view. Most user-space thread libraries work by using a predefined fixed pool of OS-level threads and scheduling coroutines on them - you can read more about building thread pools <a target="_blank" rel="noopener" href="https://www.bartoszsypytkowski.com/thread-safety-with-affine-thread-pools/">here</a>.</p>
+<h3 id="Delayed-execution"><a class="header-anchor" href="#Delayed-execution">¶</a>Delayed execution</h3>
+<p>In order to implement delays, we could theoretically just call <code>Thread.Sleep</code> and get over it, but this approach is devastating from any coroutine library point of view. Most user-space thread libraries work by using a predefined fixed pool of OS-level threads and scheduling coroutines on them - you can read more about building thread pools <a target="_blank" rel="noopener" href="https://www.bartoszsypytkowski.com/thread-safety-with-affine-thread-pools/">here</a>.</p>
 <p>However, <code>Thread.Sleep(timeout)</code> doesn’t know thread pooling mechanism - all it knows about is that we called suspending current OS thread of execution. This means, that this thread will not be awoken by kernel until timeout completes. What it means, is that none of our fibers will be able to use that thread. This is bad, because usually thread pools are made to fit in-line with number of machine CPU cores. In practice, <code>Thread.Sleep</code> may keep one of our CPU cores idle, wasting machine power in the process.</p>
 <p>For this reason we usually want to build a suspendable fibers, that will respect our thread pool. This however cannot be done without cooperation with scheduler itself. Therefore, we need to extend API of our scheduler:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">IScheduler</span> <span class="operator">=</span></span><br><span class="line">  <span class="keyword">abstract</span> Schedule<span class="operator">:</span> (<span class="type">unit</span> <span class="operator">-&gt;</span> <span class="type">unit</span>) <span class="operator">-&gt;</span> <span class="type">unit</span></span><br><span class="line">  <span class="keyword">abstract</span> Delay<span class="operator">:</span> TimeSpan <span class="operator">*</span> (<span class="type">unit</span> <span class="operator">-&gt;</span> <span class="type">unit</span>) <span class="operator">-&gt;</span> <span class="type">unit</span></span><br></pre></td></tr></table></figure>
-
 <p>And our simple implementation of it as well:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> shared <span class="operator">=</span> </span><br><span class="line">    &#123; <span class="keyword">new</span> IScheduler <span class="keyword">with</span></span><br><span class="line">      <span class="keyword">member</span> __.Schedule fn <span class="operator">=</span> <span class="operator">....</span></span><br><span class="line">      <span class="keyword">member</span> this.Delay (timeout<span class="operator">:</span> TimeSpan, fn) <span class="operator">=</span> </span><br><span class="line">        <span class="keyword">let</span> <span class="keyword">mutable</span> t <span class="operator">=</span> Unchecked.defaultof<span class="operator">&lt;</span>Timer<span class="operator">&gt;</span></span><br><span class="line">        <span class="keyword">let</span> callback <span class="operator">=</span> <span class="keyword">fun</span> _ <span class="operator">-&gt;</span> </span><br><span class="line">          t.Dispose()</span><br><span class="line">          fn()</span><br><span class="line">          ()</span><br><span class="line">        t <span class="operator">&lt;-</span> <span class="keyword">new</span> Timer(callback, <span class="literal">null</span>, int timeout.TotalMilliseconds, Timeout.Infinite)</span><br><span class="line">    &#125;</span><br></pre></td></tr></table></figure>
-
 <p>We’ll use a .NET timers here to implement our delays. With these in our hands, our<code>Fiber.delay</code> operation is trivial to implement:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> delay (timeout)<span class="operator">:</span> Fiber<span class="operator">&lt;</span><span class="type">unit</span><span class="operator">&gt;</span> <span class="operator">=</span></span><br><span class="line">  Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">    <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">    <span class="keyword">else</span> s.Delay(timeout, <span class="keyword">fun</span> () <span class="operator">-&gt;</span> </span><br><span class="line">      <span class="keyword">if</span> c.Cancelled </span><br><span class="line">      <span class="keyword">then</span> next <span class="literal">None</span> </span><br><span class="line">      <span class="keyword">else</span> next (<span class="literal">Some</span> (<span class="literal">Ok</span> ())))</span><br></pre></td></tr></table></figure>
-
-<h3 id="Composing-parallel-fibers"><a href="#Composing-parallel-fibers" class="headerlink" title="Composing parallel fibers"></a>Composing parallel fibers</h3><p>We’re slowly getting to the end. What I left for this blog post was to implement two basic operators, that are prevalent in most coroutine libraries:</p>
+<h3 id="Composing-parallel-fibers"><a class="header-anchor" href="#Composing-parallel-fibers">¶</a>Composing parallel fibers</h3>
+<p>We’re slowly getting to the end. What I left for this blog post was to implement two basic operators, that are prevalent in most coroutine libraries:</p>
 <ul>
 <li><code>Fiber.parallel</code> which will schedule multiple fibers to run in parallel and returns a fiber which aggregates their results.</li>
 <li>Running two fibers in parallel and returning the result of whichever completes first, while cancelling a second one. We already discussed this approach before. Here I’ll call it <code>Fiber.race</code>.</li>
 </ul>
-<h4 id="Aggregating-parallel-results"><a href="#Aggregating-parallel-results" class="headerlink" title="Aggregating parallel results"></a>Aggregating parallel results</h4><p>We’ll start from building a parallel operator, which will change our array of fibers into fiber with an array of results. But let’s define the semantics of that operation first:</p>
+<h4 id="Aggregating-parallel-results"><a class="header-anchor" href="#Aggregating-parallel-results">¶</a>Aggregating parallel results</h4>
+<p>We’ll start from building a parallel operator, which will change our array of fibers into fiber with an array of results. But let’s define the semantics of that operation first:</p>
 <ul>
 <li>Our result fiber completes only when all of the aggregated fibers completed with successful result.</li>
 <li>If any of the fibers fails, the resulting fiber also fails.</li>
@@ -337,32 +332,28 @@
 </ul>
 <p>The core skeleton of that operation could look like following:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> parallel (fibers<span class="operator">:</span> Fiber<span class="operator">&lt;</span><span class="symbol">&#x27;a</span><span class="operator">&gt;</span>[])<span class="operator">:</span> Fiber<span class="operator">&lt;</span><span class="symbol">&#x27;a</span>[]<span class="operator">&gt;</span> <span class="operator">=</span></span><br><span class="line">  Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">    <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">    <span class="keyword">else</span> </span><br><span class="line">      <span class="keyword">let</span> child <span class="operator">=</span> c.AddChild()</span><br><span class="line">      <span class="keyword">let</span> successes <span class="operator">=</span> Array.zeroCreate remaining</span><br><span class="line">      <span class="keyword">let</span> <span class="keyword">mutable</span> remaining <span class="operator">=</span> Array.length fibs</span><br><span class="line">      fibers <span class="operator">|&gt;</span> Array.iteri (<span class="keyword">fun</span> idx (Fiber call) <span class="operator">-&gt;</span></span><br><span class="line">        s.Schedule (<span class="keyword">fun</span> () <span class="operator">-&gt;</span> <span class="comment">(* to be defined *)</span>)</span><br><span class="line">      )</span><br></pre></td></tr></table></figure>
-
 <p>Here, we create a dedicated cancellation token, an array of results and a countdown counter - we’re going to decrement it every time one of our fibers completes to know when we’re ready to return a complete result. I’ve left a placeholder for a lambda body that we actually want to schedule. We’re going to fill it right away:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// defined above: s.Schedule &lt;| fun () -&gt;</span></span><br><span class="line">call (s, child) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span> </span><br><span class="line"><span class="keyword">match</span> result <span class="keyword">with</span></span><br><span class="line"><span class="operator">|</span> <span class="literal">Some</span> (<span class="literal">Ok</span> success) <span class="operator">-&gt;</span></span><br><span class="line">  <span class="comment">// fill the result array</span></span><br><span class="line">  successes.[idx] <span class="operator">&lt;-</span> success</span><br><span class="line">  <span class="keyword">if</span> c.Cancelled <span class="operator">&amp;&amp;</span> Interlocked.Exchange(<span class="operator">&amp;</span>remaining, <span class="number">-1</span>) <span class="operator">&gt;</span> <span class="number">0</span> <span class="keyword">then</span></span><br><span class="line">    next <span class="literal">None</span></span><br><span class="line">  <span class="keyword">elif</span> Interlocked.Decrement(<span class="operator">&amp;</span>remaining) <span class="operator">=</span> <span class="number">0</span> <span class="keyword">then</span></span><br><span class="line">    <span class="comment">// if all results have been returned, call the `next` callback</span></span><br><span class="line">    <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">    <span class="keyword">else</span> next (<span class="literal">Some</span> (<span class="literal">Ok</span> successes))</span><br><span class="line"><span class="operator">|</span> <span class="literal">Some</span> (<span class="literal">Error</span> fail) <span class="operator">-&gt;</span></span><br><span class="line">  <span class="keyword">if</span> Interlocked.Exchange(<span class="operator">&amp;</span>remaining, <span class="number">-1</span>) <span class="operator">&gt;</span> <span class="number">0</span> <span class="keyword">then</span> </span><br><span class="line">    child.Cancel() <span class="comment">// we failed, cancel other fibers</span></span><br><span class="line">    <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">    <span class="keyword">else</span> next (<span class="literal">Some</span> (<span class="literal">Error</span> fail))</span><br><span class="line"><span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span></span><br><span class="line">  <span class="keyword">if</span> Interlocked.Exchange(<span class="operator">&amp;</span>remaining, <span class="number">-1</span>) <span class="operator">&gt;</span> <span class="number">0</span> <span class="keyword">then</span> next <span class="literal">None</span>)</span><br></pre></td></tr></table></figure>
-
 <p>As you probably noticed, we’re using <code>Interlocked</code> class again - that’s because now we have multiple fibers running in parallel, therefore our access to shared mutable values is not thread safe. This includes <code>remaining</code> counter decrement operation. This however doesn’t apply to <code>successes.[i] &lt;- success</code> - since every fiber knows and touches only its own index within result array, there’s no worry that any other will try to push its result in the same place.</p>
-<p>What you also can see, we’re using a <code>-1</code> here as a magic value - we’ll use it on the counter as a flag to determine if any of the fibers failed&#x2F;was cancelled - and if so, which one of them will call the <code>next</code> callback.</p>
-<h4 id="Racing-to-completion"><a href="#Racing-to-completion" class="headerlink" title="Racing to completion"></a>Racing to completion</h4><p>With first operator (<code>Fiber.parallel</code>) ready, now it’s the time to implement <code>Fiber.race</code>. Since I’ve discussed it behavior multiple times in this post already, let’s dive straight into the code:</p>
+<p>What you also can see, we’re using a <code>-1</code> here as a magic value - we’ll use it on the counter as a flag to determine if any of the fibers failed/was cancelled - and if so, which one of them will call the <code>next</code> callback.</p>
+<h4 id="Racing-to-completion"><a class="header-anchor" href="#Racing-to-completion">¶</a>Racing to completion</h4>
+<p>With first operator (<code>Fiber.parallel</code>) ready, now it’s the time to implement <code>Fiber.race</code>. Since I’ve discussed it behavior multiple times in this post already, let’s dive straight into the code:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> race (Fiber left) (Fiber right)<span class="operator">:</span> Fiber<span class="operator">&lt;</span>Choice<span class="operator">&lt;</span><span class="symbol">&#x27;a</span>, <span class="symbol">&#x27;b</span><span class="operator">&gt;&gt;</span> <span class="operator">=</span></span><br><span class="line">  Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">    <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">    <span class="keyword">else</span> </span><br><span class="line">      <span class="keyword">let</span> <span class="keyword">mutable</span> flag <span class="operator">=</span> <span class="number">0</span></span><br><span class="line">      <span class="keyword">let</span> cancelChild <span class="operator">=</span> c.AddChild()</span><br><span class="line">      <span class="keyword">let</span> run fiber choice <span class="operator">=</span></span><br><span class="line">          <span class="comment">(* to be described *)</span></span><br><span class="line">      run left Choice1Of2</span><br><span class="line">      run right Choice2Of2</span><br></pre></td></tr></table></figure>
-
 <p>So again, we want to have shared mutable flag, which we’ll use to determine, which of the fibers finished as a first one to be able to call fiber’s callback safely and cancel the other. You may see, that our returned fiber uses <code>Choice&lt;,&gt;</code> type - this means, that our left and right fibers can have results of different types. We’ll use that soon, but first we need to complete our <code>run</code> function body:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> run fiber choice <span class="operator">=</span></span><br><span class="line">  s.Schedule (<span class="keyword">fun</span> () <span class="operator">-&gt;</span></span><br><span class="line">    fiber (s, cancelChild) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span></span><br><span class="line">      <span class="keyword">if</span> Interlocked.Exchange(<span class="operator">&amp;</span>flag, <span class="number">1</span>) <span class="operator">=</span> <span class="number">0</span> <span class="keyword">then</span></span><br><span class="line">        cancelChild.Cancel()</span><br><span class="line">        <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">        <span class="keyword">else</span> <span class="keyword">match</span> result <span class="keyword">with</span></span><br><span class="line">             <span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span> next <span class="literal">None</span></span><br><span class="line">             <span class="operator">|</span> <span class="literal">Some</span>(<span class="literal">Ok</span> v) <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Ok</span>(choice v)))</span><br><span class="line">             <span class="operator">|</span> <span class="literal">Some</span>(<span class="literal">Error</span> e) <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Error</span> e))))</span><br></pre></td></tr></table></figure>
-
 <p>What we do here is simply trying to race to “reserve” out flag variable - the winner gets his result mapped to corresponding choice, while looser gets cancelled.</p>
 <p>What’s interesting, we can now combine our <code>race</code> and <code>delay</code> functions to easily implement timeout mechanism:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> timeout (t<span class="operator">:</span> TimeSpan) fiber <span class="operator">=</span></span><br><span class="line">  Fiber <span class="operator">&lt;|</span> <span class="keyword">fun</span> (s, c) next <span class="operator">-&gt;</span></span><br><span class="line">    <span class="keyword">let</span> (Fiber call) <span class="operator">=</span> race (delay t) fiber</span><br><span class="line">    call (s, c) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span></span><br><span class="line">      <span class="keyword">if</span> c.Cancelled <span class="keyword">then</span> next <span class="literal">None</span></span><br><span class="line">      <span class="keyword">else</span> <span class="keyword">match</span> result <span class="keyword">with</span></span><br><span class="line">           <span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span> next <span class="literal">None</span></span><br><span class="line">           <span class="operator">|</span> <span class="literal">Some</span>(<span class="literal">Ok</span> (Choice1Of2 _)) <span class="operator">-&gt;</span> next <span class="literal">None</span> <span class="comment">// timeout won</span></span><br><span class="line">           <span class="operator">|</span> <span class="literal">Some</span>(<span class="literal">Ok</span> (Choice2Of2 v)) <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Ok</span> v))</span><br><span class="line">           <span class="operator">|</span> <span class="literal">Some</span>(<span class="literal">Error</span> e) <span class="operator">-&gt;</span> next (<span class="literal">Some</span>(<span class="literal">Error</span> e))</span><br><span class="line">    )</span><br></pre></td></tr></table></figure>
-
 <p>The one last thing left for us, is to be able to run out fibers on the main thread - otherwise we’d start our program, schedule fibers to run in the background and then close the program without waiting for the results.</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> blocking (s<span class="operator">:</span> IScheduler) (cancel<span class="operator">:</span> Cancel) (Fiber fn) <span class="operator">=</span></span><br><span class="line">  <span class="keyword">use</span> waiter <span class="operator">=</span> <span class="keyword">new</span> ManualResetEventSlim(<span class="literal">false</span>)</span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> res <span class="operator">=</span> <span class="literal">None</span></span><br><span class="line">  s.Schedule(<span class="keyword">fun</span> () <span class="operator">-&gt;</span> fn (s, cancel) (<span class="keyword">fun</span> result <span class="operator">-&gt;</span></span><br><span class="line">    <span class="keyword">if</span> <span class="built_in">not</span> cancel.Cancelled <span class="keyword">then</span></span><br><span class="line">      Interlocked.Exchange(<span class="operator">&amp;</span>res, <span class="literal">Some</span> result) <span class="operator">|&gt;</span> <span class="built_in">ignore</span></span><br><span class="line">    waiter.Set()))</span><br><span class="line">  waiter.Wait()</span><br><span class="line">  res.Value</span><br></pre></td></tr></table></figure>
-
 <p>It’s simple - we’ll use standard synchronization primitives provided by .NET runtime, to hold current OS thread until we complete. Sure it’s blocking an OS thread, but we’ll eventually need that if we don’t want our program’s main function to finish before all fibers inside the thread pool complete.</p>
-<h3 id="Simulating-real-environment-in-tests"><a href="#Simulating-real-environment-in-tests" class="headerlink" title="Simulating real environment in tests"></a>Simulating real environment in tests</h3><p>In theory, we could be done here. But, if you managed to read up to this point, we may want to cover one last scenario. Imagine that we’d want to test our fibers. However running tests using standard thread pool scheduler can lead to funky issues:</p>
+<h3 id="Simulating-real-environment-in-tests"><a class="header-anchor" href="#Simulating-real-environment-in-tests">¶</a>Simulating real environment in tests</h3>
+<p>In theory, we could be done here. But, if you managed to read up to this point, we may want to cover one last scenario. Imagine that we’d want to test our fibers. However running tests using standard thread pool scheduler can lead to funky issues:</p>
 <ul>
 <li>Sometimes you may trigger some race conditions in your code, that only happen in specific situations (like high CPU contention) and are almost impossible to reproduce during debug sessions.</li>
-<li>Other times you may have some lengthy delays&#x2F;timeouts in your code, like waiting for seconds or even minutes before continuing. Guess what: now your test will wait for just as long.</li>
+<li>Other times you may have some lengthy delays/timeouts in your code, like waiting for seconds or even minutes before continuing. Guess what: now your test will wait for just as long.</li>
 </ul>
-<p>These are not new problems. They are well known in world of concurrent and distributed systems. What we need, is a simulation of execution environment. If you want to listen more about that concept, I could recommend you <a target="_blank" rel="noopener" href="https://www.youtube.com/watch?v=4fFDFbi3toc&ref=bartoszsypytkowski.com">this presentaton</a>. To run our test predictably, we’ll create a dedicated test scheduler, which will run our code in deterministic fashion (on a single core) and in a way that’s detached from other invariants eg. actual physical clock and random number generator.</p>
+<p>These are not new problems. They are well known in world of concurrent and distributed systems. What we need, is a simulation of execution environment. If you want to listen more about that concept, I could recommend you <a target="_blank" rel="noopener" href="https://www.youtube.com/watch?v=4fFDFbi3toc&amp;ref=bartoszsypytkowski.com">this presentaton</a>. To run our test predictably, we’ll create a dedicated test scheduler, which will run our code in deterministic fashion (on a single core) and in a way that’s detached from other invariants eg. actual physical clock and random number generator.</p>
 <p>The idea here is simple - our scheduler will operate on notion of virtual timeline. When we’ll try to schedule a new function - to trigger either immediately or after some timeout - we’ll store it inside an ordered collection, a timeline. Some of the technical decisions we also made for purposes of this implementation:</p>
 <ul>
 <li>Whenever a fiber is going to schedule multiple parallel executions “at the same time”, we’ll put them all into a single bucket on a timeline. Later on I’ll cover, why this is useful.</li>
@@ -370,20 +361,16 @@
 </ul>
 <p>After describing the concept behind the algorithm, the actual implementation really shouldn’t be that surprising:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">TestScheduler</span>(now<span class="operator">:</span> DateTime) <span class="operator">=</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> running <span class="operator">=</span> <span class="literal">false</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> currentTime <span class="operator">=</span> now.Ticks</span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> timeline <span class="operator">=</span> Map.empty</span><br><span class="line">  <span class="keyword">let</span> schedule delay fn <span class="operator">=</span> <span class="comment">(* to be defined *)</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">rec</span> run () <span class="operator">=</span> <span class="comment">(* to be defined *)</span></span><br><span class="line">  <span class="keyword">interface</span> IScheduler <span class="keyword">with</span></span><br><span class="line">    <span class="keyword">member</span> this.Schedule fn <span class="operator">=</span> </span><br><span class="line">      schedule <span class="number">0</span>L fn</span><br><span class="line">      <span class="keyword">if</span> <span class="built_in">not</span> running <span class="keyword">then</span></span><br><span class="line">        running <span class="operator">&lt;-</span> <span class="literal">true</span></span><br><span class="line">        run ()</span><br><span class="line">    <span class="keyword">member</span> this.Delay (timeout<span class="operator">:</span> TimeSpan, fn) <span class="operator">=</span> schedule timeout.Ticks fn</span><br></pre></td></tr></table></figure>
-
 <p>We’re using a <code>running</code> flag here to not try to invoke <code>run</code> multiple times in nested manner - this would cause non-tailable recursion and potential stack overflow in more expensive tests.</p>
 <p>The schedule function is pretty simple - calculate expected execution time for a function, then add that function to be executed at that point in time.</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> schedule delay fn <span class="operator">=</span> </span><br><span class="line">  <span class="keyword">let</span> at <span class="operator">=</span> currentTime <span class="operator">+</span> delay</span><br><span class="line">  timeline <span class="operator">&lt;-</span></span><br><span class="line">    <span class="keyword">match</span> Map.tryFind at timeline <span class="keyword">with</span></span><br><span class="line">    <span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span> Map.add at [fn] timeline </span><br><span class="line">    <span class="operator">|</span> <span class="literal">Some</span> fns <span class="operator">-&gt;</span> Map.add at (fn<span class="operator">::</span>fns) timeline</span><br></pre></td></tr></table></figure>
-
 <p>Given all of the code we already survived in this blog post, run loop should be pretty simple:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="keyword">rec</span> run () <span class="operator">=</span></span><br><span class="line">  <span class="keyword">match</span> Seq.tryHead timeline <span class="keyword">with</span></span><br><span class="line">  <span class="operator">|</span> <span class="literal">None</span> <span class="operator">-&gt;</span> running <span class="operator">&lt;-</span> <span class="literal">false</span></span><br><span class="line">  <span class="operator">|</span> <span class="literal">Some</span> (KeyValue(time, bucket)) <span class="operator">-&gt;</span></span><br><span class="line">    timeline <span class="operator">&lt;-</span> Map.remove time timeline</span><br><span class="line">    currentTime <span class="operator">&lt;-</span> time</span><br><span class="line">    <span class="keyword">for</span> fn <span class="keyword">in</span> List.rev bucket <span class="keyword">do</span> </span><br><span class="line">      fn ()          </span><br><span class="line">    run ()</span><br></pre></td></tr></table></figure>
-
 <p>We’ll try to pick the first entry from the timeline - since here we use F# map, which is sorted in ascending order, we know that first entry is the one with the shortest execution timeout. We update our “current” time to match the expected one we calculated earlier, and finally we execute all functions scheduled at that time and repeat the loop all over until we eventually run out of scheduled actions.</p>
 <p>Now here’s the trick - we use <code>List.rev</code> to execute functions in the same order in which they were scheduled, because we want our tests to be deterministic and our bugs to be reproducible. However this is not the only strategy - since we know that functions in the same bucket could as well be executing in parallel, we could shuffle them around in different permutations for early discovery of some data races! I’ll won’t dive into it, but leave that idea as food for thoughts for you.</p>
 <p>One last note about the test scheduler is that isolating it from the actual physical clock means, we cannot trust our time functions (like <code>DateTime.UtcNow</code>) any longer. This shouldn’t really be an issue though - because relying on physical time would potentially make our tests indeterministic, we didn’t want to use it anyway, right?</p>
 <p>However, we need to be able to obtain current time from the scheduler, so we need to extend its API:</p>
 <figure class="highlight fsharp"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> <span class="title class_">IScheduler</span> <span class="operator">=</span></span><br><span class="line">  <span class="keyword">abstract</span> UtcNow<span class="operator">:</span> <span class="type">unit</span> <span class="operator">-&gt;</span> <span class="type">unit</span></span><br><span class="line">  <span class="comment">// ... other methods</span></span><br><span class="line">  </span><br><span class="line"><span class="keyword">type</span> <span class="title class_">TestScheduler</span>() <span class="operator">=</span></span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mutable</span> currentTime <span class="operator">=</span> DateTime.UtcNow.Ticks</span><br><span class="line">  <span class="comment">// ... rest of the implementation</span></span><br><span class="line">  <span class="keyword">interface</span> IScheduler <span class="keyword">with</span></span><br><span class="line">    <span class="keyword">member</span> __.UtcNow() <span class="operator">=</span> DateTime(currentTime)</span><br><span class="line">    <span class="comment">// ... other methods</span></span><br></pre></td></tr></table></figure>
-
 <p>And that’s all. As always, if you got confused or have a problems along the way, you can get the entire code <a target="_blank" rel="noopener" href="https://gist.github.com/Horusiath/9c790691130150b524aaa9ab426ed982?ref=bartoszsypytkowski.com">here</a>. I wanted to thank to Anthony Lloyd for his initial work on porting Scala <a target="_blank" rel="noopener" href="https://zio.dev/?ref=bartoszsypytkowski.com">ZIO</a> library to F#, which brought me an inspiration to write this piece.</p>
 
 </div> 
-- 
cgit v1.2.3