worker thread library recommendations - Haskell

Welcome to the Functional Programming Zulip Chat Archive. You can join the chat here.

Pavan Rikhi

I've got a servant web server that I want to add async worker threads for, so that routes can return early instead of waiting for an email to send or an image to be optimized.

I've got a persistent database, so I was thinking of using that as a queue, something likeTaskQueue { action :: Task, queuedAt :: UTCTime, runAt :: Maybe UTCTime, retries :: Int }

For the workers I'm thinking of using this article for inspiration: https://hackernoon.com/assume-it-worked-and-fix-later-8436d18b7ed3
But there's seems to be a handful of libraries that I could use for the thread management:
https://hackage.haskell.org/package/supervisors-0.2.0.0
https://hackage.haskell.org/package/threads-supervisor-1.2.0.1
https://hackage.haskell.org/package/slave-thread
https://hackage.haskell.org/package/immortal

Wondering if there's a common library that gets used for this or would be better suited to my usecase

During account signup, a web server will make an HTTP request to send an email. Not only are synchronous requests slow, but if the remote host is unresponsive, the application can become unresponsive.
Ben Kolera

I don't know of any sadly, but if you find one lmk. I'd really love to see a library to wrap up some CQRS style patterns. There seemed to be a few attempts at starting Event Sourcing libs a few years ago, but I think that they all died sadly.

Pavan Rikhi

I never found one but I hacked something together with MVars, Async, and Immortal.
I haven't done much concurrency work but it doesn't deadlock & it lets my routes return faster so... good enough for me :P

Generic module:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/ImmortalQueue.hs
Implementation pulling tasks from my DB:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/Workers.hs

I wonder if that's a decent enough API for something like this. Tempted to figure out how to test it and stick it in a package.

Southern Exposure's E-Commerce Website, Built with Haskell & Elm. - Southern-Exposure-Seed-Exchange/southernexposure.com
Southern Exposure's E-Commerce Website, Built with Haskell & Elm. - Southern-Exposure-Seed-Exchange/southernexposure.com
IC Rainbow

Nice. But you have a race in assignWork - between tryReadMVar and putMVar a few calls later.

Pavan Rikhi

Hmm, do you mean something might put something in the MVar between my tryReadMVar call & the putMVar call?

Pavan Rikhi

Also, I appreciate you reviewing the code :heart_eyes:

Pavan Rikhi

(that Just workerData branch in assignWork is the only thing that ever puts actions into the wdInputMVar)

Pavan Rikhi

Finally coming back to this, if anyone's got a couple minutes I'd love a review before I fork this out into a separate package:
https://github.com/Southern-Exposure-Seed-Exchange/southernexposure.com/blob/develop/server/src/ImmortalQueue.hs

Southern Exposure's E-Commerce Website, Built with Haskell & Elm. - Southern-Exposure-Seed-Exchange/southernexposure.com