I've got a servant web server that I want to add async worker threads for, so that routes can return early instead of waiting for an email to send or an image to be optimized.
I've got a persistent database, so I was thinking of using that as a queue, something likeTaskQueue { action :: Task, queuedAt :: UTCTime, runAt :: Maybe UTCTime, retries :: Int }
During account signup, a web server will make an HTTP request to send an email. Not only are synchronous requests slow, but if the remote host is unresponsive, the application can become unresponsive.
I don't know of any sadly, but if you find one lmk. I'd really love to see a library to wrap up some CQRS style patterns. There seemed to be a few attempts at starting Event Sourcing libs a few years ago, but I think that they all died sadly.
I never found one but I hacked something together with MVars, Async, and Immortal.
I haven't done much concurrency work but it doesn't deadlock & it lets my routes return faster so... good enough for me :P
I've got a servant web server that I want to add async worker threads for, so that routes can return early instead of waiting for an email to send or an image to be optimized.
I've got a persistent database, so I was thinking of using that as a queue, something like
TaskQueue { action :: Task, queuedAt :: UTCTime, runAt :: Maybe UTCTime, retries :: Int }
For the workers I'm thinking of using this article for inspiration: https://hackernoon.com/assume-it-worked-and-fix-later-8436d18b7ed3
But there's seems to be a handful of libraries that I could use for the thread management:
https://hackage.haskell.org/package/supervisors-0.2.0.0
https://hackage.haskell.org/package/threads-supervisor-1.2.0.1
https://hackage.haskell.org/package/slave-thread
https://hackage.haskell.org/package/immortal
Wondering if there's a common library that gets used for this or would be better suited to my usecase
I don't know of any sadly, but if you find one lmk. I'd really love to see a library to wrap up some CQRS style patterns. There seemed to be a few attempts at starting Event Sourcing libs a few years ago, but I think that they all died sadly.
I never found one but I hacked something together with MVars, Async, and Immortal.
I haven't done much concurrency work but it doesn't deadlock & it lets my routes return faster so... good enough for me :P
Generic module:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/ImmortalQueue.hs
Implementation pulling tasks from my DB:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/Workers.hs
I wonder if that's a decent enough API for something like this. Tempted to figure out how to test it and stick it in a package.
Nice. But you have a race in
assignWork
- betweentryReadMVar
andputMVar
a few calls later.Hmm, do you mean something might put something in the MVar between my
tryReadMVar
call & theputMVar
call?Also, I appreciate you reviewing the code :heart_eyes:
(that
Just workerData
branch inassignWork
is the only thing that ever puts actions into thewdInputMVar
)Finally coming back to this, if anyone's got a couple minutes I'd love a review before I fork this out into a separate package:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/ImmortalQueue.hs