Either vs Error - Polysemy

So I have this effect:

data FooAPI m where
  getFooBar :: Text -> FooAPI m (Either Text Bar)

Can I make getFooBar require the Member (Error Text) constraint, such that it doesn't have to return Either? Or is this an anti-pattern for polysemy? If so, what would be the recommended approach?

that also has been my most substantial coherence concern so far

haven't done any meaningful research though

you can catch an error blindly from a consumer, but that's not very reasonable

I personally prefer to use the Error effect and not be specific about the returning type being able to "fail". I start by being explicit tho and then refine the signatures I don't see any need to be explicit

I have been wondering about this too. In a work I have in progress I have left the error constraint out of the API but have it in the interpreter. This way you could have an interpreter in which getFooBar always returns a Bar, and never throws an error. But it also means that you still have to handle the errors even if you only use API functions that don't throw errors.

I use Either in all effect APIs and call fromEither on every use. So Error is only used in the intermediate machinery, converting to/from Either on the interpreter boundaries. It would be really nice to have something to simplify that.

my preference tbh would be to build the error type into Sem

like zio does over in scala world

don't know any specifics there tho

probably also complicated to make this flexible, i.e. keep a handled error from propagating in the signature

one possibility would be a TH constructor that automates the fromEither call

but all of this is indicative of the main problem: the error is inherently specific to an interpreter, so exposing it as an Either signature is already premature.

this was going to be my original response too - there's no need to put it in the effect itself, but then I realised there might be some effects that always have the inherent possibility to fail, and handling their failure is also a part of your "business logic" (?)

Like API errors

exactly, I guess there is a substantial conceptual difference between using Error in a oneshot CLI app where you would always terminate the whole process on error and something daemonic like a web app.

You could argue that this is inherently an abuse of Error, I guess

so what you would need is the Polysemy machinery checking whether an Error is caught somewhere in the other interpreters (or the main program), though that would need them to know about the errors again. you could do something like making all errors that are interpreter-specific into Text and catching that as a NonFatal X where X is the effect whose interpreter throws

could be an effect CatchNonFatal you use in the consumer that is interpreted as interpretCNF :: Sem (CatchNonFatal X : NonFatal X InterpreterError : Error InterpreterError : r) a-> Sem r a

if the consumer needs a data type for the error, then it's obviously something non-specific and may be part of the API. I do sometimes use data type errors specific to an interpreter to decide on a consequence in a consumer, so this would enforce some discipline in that area

I'd just do an indirection:

data FooAPI m where
  GetFooBar' :: Text -> FooAPI m (Either Text Bar)

getFooBar
  :: Members [FooAPI, Error Text] r
  => Text
  -> Sem r Bar
getFooBar str = getFooBar' str >>= fromEither

Or is there some reason that wouldn't work?

Sorry, just noticed that's what Torsten suggested

@Love Waern (King of the Homeless) so what's your opinion on whether this is a sane use case for Error?

If the exception is inherently part of the effect, so that you must have the user or want to enforce the user to deal with the exception, then yes, this is a good way to go about it.
An alternative way is to instead do the conversion at the interpreter level.
Take the example of, say, a REST api that you can do for something stateful that might fail:

getRestfulState :: IO (Either RestfulException RestfulState)
getRestfulState = ...

putRestfulState :: s -> IO (Either RestfulException RestfulState)
putRestfulState s = ...

runRestfulState
  :: (Member IO r, Member (Error RestfulException) r)
  => Sem (State RestfulState ': r) a
  -> Sem r a
runRestfulState = interpret $ \case
  Put s -> fromEitherM (putRestfulState s)
  Get -> fromEitherM getRestfulState

That way, users can use get of the State RestfulState effect, and have it raise an exception, without the Error RestfulException effect being in scope.

You want the ability to deal with the exception in application code? No problem. Just have an Member (Error RestfulException) r at the place where you want to deal with the exception, and then use catch. Notably, you can do this without needing Member (Embed IO) r like the interpreter does.

There is a problem however: you can't, out-of-the-box, have more granular exceptions. Say you want to deal with aRestfulException generated by put and get, but in a granular way: in terms of an Error effect that's smaller than Error RestfulException, or maybe not even connected to Rest at all: say,Error PutException. If so, then you need an interpreter later down the line to convert Error PutException to Error RestfulException. You might reach for mapError, but NOPE, it will actually break due to #225. I hadn't actually realized this until now. The example in #225 is contrived in comparison.

But the following will fix that:

errorToError
  :: Member (Error e2) r
  => (e1 -> e2)
  -> (e2 -> Maybe e1)
  -> Sem (Error e1 ': r) a
  -> Sem r a
errorToError to from = interpretH $ \case
  Throw e -> throw (to e)
  Catch m h -> do
    m' <- errorToError to from <$> runT m
    h' <- (errorToError to from .) <$> bindT h
    s  <- getInitialStateT
    raise $ catch m' $ \e -> case from e of
      Just e' -> h' (e' <$ s)
      _       -> throw e

Which I am now convinced should get added to Polysemy v2.0, probably together with a deprecation of mapError. For the moment, stick that into your code, and use it to convert between your Errors.
This way, you can get:

1. The choice to be ignorant to the fact that an action of an effect may cause an exception generated by the interpreter for that effect
2. The ability to handle exceptions in a granular way when you do care about it

WASN'T DONE. HIT ENTER BY ACCIDENT

Now done.

Actually, now when I think about it, sometimes conversions only go one way. For example, it might make sense to convert RestfulException to Maybe PutException, but not PutException to RestfulException. In that case, what you really want is an Error effect without throw to use in application code, à-la fused-effects. That's something we'd need to think more about.

my concern with catch is that the compiler won't notify me that I forgot to handle the interpreter's error in my program since using catch doesn't remove the need for the Error interpreter after the throwing interpreter. Obviously this isn't exactly canonical for Error, but it's what happens to me a lot. I was wondering if I'm just doing things wrong.

Hmm. Then what about combining the indirection approach with a pseudo-action that you can use in place of catch that does give you that property, like this:

catch' :: Sem (Error e ': r) a -> (e -> Sem r a) -> Sem r a
catch' m h = runError m >>= either h pure

Then you can do this:

data FooAPI m where
  GetFooBar' :: Text -> FooAPI m (Either Text a)

getFooBar
  :: Members '[FooAPI, Error Text] r
  => Text
  -> Sem r Bar
getFooBar str = getFooBar' str >>= fromEither

-- Compiles:
one :: Members '[FooAPI r, Error Text] => Sem r (Text, Text)
one = do
  t1 <- getFooBar
  t2 <- getFooBar
  return (t1, t2)

two :: Members '[FooAPI r, Error Text] => Sem r (Text, Text)
two = do
  t1 <- getFooBar `catch'` \e -> return $ "Bad1: " <> e
  t2 <- getFooBar
  return (t1, t2)

usingTwo :: Member FooApi r => Sem r Text
usingTwo = (`catch'` \e -> return $ "Bad12:" <> e) $ do
  (t1, t2) <- two
  return $ t1 <> t2

three :: Member FooAPI r => Sem r (Text, Text)
three = do
  t1 <- getFooBar `catch'` \e -> return $ "Bad1: " <> e
  t2 <- getFooBar `catch'` \e -> return $ "Bad2: " <> e
  return (t1, t2)

-- Doesn't compile due to the second use of getFooBar:
bad :: Member FooAPI r => Sem r (Text, Text)
bad = do
  t1 <- getFooBar `catch'` \e -> return $ "Bad1: " <> e
  t2 <- getFooBar
  return (t1, t2)

Hey, isn't this a trick some first-order extensible effects library used to get pseudo-error? Since with this, you don't need an higher-order effect for catch; you can use the pseudo-action instead.

Yeah, freer-simple!

There are probably some issues with this: I think it messes up local/global state semantics, since you're effectively taking a global "Error" effect and making it local. I'd need to look into it some more.

But all things considered, I quite like this. Shame you can't call it catch, though.
... But maybe we can call it handle, if we decide to add it.

I'm confused, you have the Error constraint on getFooBar now visible from the programs one etc?

With catch', the error constraint is visible as long as there exists some getFooBar the possible exception of which hasn't been handled.
Compare with three and usingTwo; they don't have the constraint.

my mind is going in circles now.

I was starting from this place when talking about catch:

data E = E deriving Show

data A :: Effect where
  DoA :: A m ()

makeSem ''A

prog ::
  Member (Error E) r =>
  Member A r =>
  Sem r ()
prog =
  catch doA (\ _ -> pure ())

interpretA ::
  Member (Error E) r =>
  InterpreterFor A r
interpretA =
  interpret $ \ DoA -> throw E

so here the fact that E is thrown by doAis not visible to prog

but it is also an error specific to interpretA and might not be thrown by a different interpreter, and moreover the different one might throw something else

so my earlier suggestion was to have an additional AError Text that is used as a something went wrong in A marker, which is then caught in prog and converted from E at the interpretation point

and then my question was "how do I ensure that this error is handled in prog"

Ok, I get it. Let me think a bit.

so this might be a completely unreasonable way to approach the whole subject but it's what transpired a lot for me

my earlier suggestion was to have an additional error effect that is used only for the purpose of connecting the two, as a constraint on prog

let my try to implement this quick

I came up with this evil piece of magic:

data AManager m a where
  ManageA :: (Member (Error E) r, Member AManager r) => Sem (A ': r) a -> AManager (Sem r) a

manageA :: (Member (Error E) r, Member AManager r) => Sem (A ': r) a -> Sem r a
manageA = send . ManageA

prog ::
  Member AManager r =>
  Sem r ()
prog = catch' (manageA doA) (\ _ -> pure ())


interpretA
  :: InterpreterFor (AManager (Sem r)) r
interpretA = interpretH $ \(ManageA m) -> do
  m' <- runT $ (`interpret` m) $ \DoA -> throw E
  raise (interpretA m')

But this is pretty damn ugly and bad

Wait, hold on... this ain't enough.

:slight_smile:

This is gonna get so damn ugly...

already interesting to see a HO effect specialized to Sem r interpreted :smiley:

Going back to the FooAPI example, replacing Error Text with Error SmallExc.

data FooAPI m where
  GetFooBar :: Text -> FooAPI m a

makeSem ''FooAPI

data FooAPIManager c e m a where
  ManageFooAPI :: forall c e r a. Members '[FooAPIManager c e, Error e] r => Sem (FooAPI ': r) a -> FooAPIManager c e (Sem r) a
  ManageFooAPICarrier :: forall c e m a. c a -> FooAPIManager c e m a

manageFooAPI  :: forall c e r a. (Member e r, Member (FooAPIManager c e r) => Sem (FooAPI ': r) a -> Sem r a
manageFooAPI  = send . ManageFooAPI @c @e

-- Some progam that makes use of getFooBar without knowing or caring that it might cause exceptions
prog' :: Member FooAPI r => Sem r ()
prog' = getFooBar "hiho"

-- Running that piece of the program in the context of the larger application.
prog ::
  Member (FooAPIManager c SmallExc) r =>
  Sem r ()
prog = catch' (manageFooAPI  @c @SmallExc prog') (\ _ -> pure ())

-- The primitive behind getFooBar
getFooBarIO :: IO (Either BigExc a)
getFooBarIO = ...

bigToSmall :: BigExc -> Maybe SmallExc
bigToSmall = ...

runFooAPIManager :: Members '[Embed IO r, Error BigExc] r => InterpreterFor (FooAPIManager (Sem r) SmallExc ': r)
runFooAPIManager = interpretH $ \case
  ManageFooAPI m -> do
    m' <- runT $ (`interpret` m) $ \(GetFooBar exc) -> do
      e <- send $ ManageFooAPICarrier $ tryJust bigToSmall (fromEitherM getFooBarIO)
      fromEither e
    raise (runFooAPIManager m')
  ManageFooAPICarrier c -> raise c >>= pureT

This is terrible, but it works, somehow.
FooAPIManager is the "link", here. You can use getFooBarin a part of the program without knowing it throws exceptions, but eventually, in order to run that part, you must use manageFooAPI to convert it, at which point any exception makes itself known, and you must handle that exception by using catch'. FooApiManager has an additional type variable for the carrier, which is needed to actually interpret GetFooBar with the parametrized Sem r0 seen inside ManageFooAPI, since we don't know anything about it other than it carries FooAPIManager (Sem r) SmallExc and Error SmallExc. By keeping that type variable polymorphic inside user code, you keep user code from doing naughty things with it.
I also show here how you can do this in a way that exposes a smaller, granular exception to the user code rather than the bigger exception that the primitive may generate.

This is complicated, evil, and very, very ugly. I'll try to find some way to improve it.

Keep in mind I haven't checked if this compiles. It should work morally.

phew, I'll have to study that for a while

I haven't read this entire discussion, but it seems to me that I should just stick to returning Eithers

@Love Waern (King of the Homeless) can you tell me why this doesn't work? I'm trying to understand the semantics of throwing and catching.

data NonFatal e er =
  NonFatal er
  deriving Show

data CatchNonFatal e =
  CatchNonFatal Text
  deriving Show

data E = E deriving Show

data A :: Effect where
  DoA :: A m Int

makeSem ''A

data NF a =
  NF Text
  deriving Show

data CNF e :: Effect where
  ThrowNF :: Text -> CNF e m a
  CatchNF :: m a -> (Text -> m a) -> CNF e m a

makeSem ''CNF

intCNF ::
  Member (Error (NF A)) r =>
  InterpreterFor (CNF A) r
intCNF =
  interpretH $ \case
    ThrowNF er -> throw (NF er)
    CatchNF ma recover ->
      (catch @(NF A) (run ma) (\ (NF er) -> run (recover er)))
      where
        run =
          raise . intCNF <=< runT

run ::
  Member (CNF A) r =>
  Member A r =>
  Sem r Int
run =
  catchNF doA $ \ _ -> pure 1

interpretA ::
  Member (Error E) r =>
  InterpreterFor A r
interpretA =
  interpret $ \ DoA -> throw E

intNF ::
  Member (CNF A) r =>
  Sem (Error (NonFatal A E) : r) a ->
  Sem r a
intNF =
  cnf <=< runError
  where
    cnf (Left (NonFatal er)) = throwNF (show er)
    cnf (Right a) = pure a

test_errorBad :: IO ()
test_errorBad =
  print =<< (
    runFinal $
      runError $
      intCNF $
      intNF $
      mapError (NonFatal @A @E) $
      interpretA $
      run
  )

the throw in intCNF is caught by runError in the main function, not catchNF.

Because mapError is broken.

See #225

It's only just now I realized the extent of how horrible it is

Like I said above, use errorToError (which isn't in polysemy yet) instead

nope, same with errorToError :speechless:

Oh, sorry. It's probably becauseintNF converts its argument as a whole, rather than individual throws. Same problem as with mapError. That's also fixed by defining it in terms of errorToError.

My suggestion for mapErrorVia in #225 has the same problem, so that's quite stupid of me in hindsight.

ugh, obviously when I use runError, an X.catch is inserted :smiley: now it works!

even with mapError

Cooked up an alternative to that manager garbage.

import Polysemy
import Polysemy.Internal
import Polysemy.Internal.Union

import Polysemy.Error

data Exceptional err eff m a where
  Exceptionally
    :: forall err eff r0 a
     . Weaving eff (Sem r0) a
    -> Exceptional err eff (Sem r0) (Either err a)

-- Run a Sem using actions of `eff`if you're prepared to deal with any exception that may arise from those actions.
exceptionally
  :: forall err eff r a
   . Members '[Exceptional err eff, Error err] r
  => Sem (eff ': r) a
  -> Sem r a
exceptionally sem = Sem $ \k -> runSem sem $ \u -> case decomp (hoist (exceptionally @err) u) of
  Right wav -> do
    runSem (send (Exceptionally @err wav) >>= fromEither) k
  Left g -> k g

-- Runs `Exceptional exc eff` if `eff`is in the effect stack, given a way to convert from a larger exception to `exc`.
runExceptional
  :: forall eff smallExc bigExc r a
   . Members '[eff, Error bigExc] r
  => (bigExc -> Maybe smallExc)
  -> Sem (Exceptional smallExc eff ': r) a
  -> Sem r a
runExceptional from sem = Sem $ \k -> runSem sem $ \u -> case decomp (hoist (runExceptional from) u) of
  Right (Weaving (Exceptionally wav) s wv ex ins) ->
    let
      main = liftSem $ weave s wv ins (injWeaving wav)
    in
      (`runSem` k) $
        catchJust from (fmap (ex . fmap Right) main) $ \e -> pure (ex (Left e <$ s))
  Left g -> k g

handle :: Sem (Error e ': r) a -> (e -> Sem r a) -> Sem r a
handle m h = runError m >>= either h pure

-- Boilerplate ends here.

data FooAPI m a where
  GetFooBar :: String -> FooAPI m a

makeSem ''FooAPI

data BigExc
data SmallExc

-- Some progam that makes use of getFooBar without knowing or caring that it might cause exceptions
prog' :: Member FooAPI r => Sem r ()
prog' = getFooBar "hiho"

-- Running that piece of the program in the context of the larger application.
prog ::
  Member (Exceptional SmallExc FooAPI) r =>
  Sem r ()
prog = handle @SmallExc (exceptionally @SmallExc @FooAPI prog') (\ _ -> pure ())

-- The primitive behind getFooBar
getFooBarIO :: String -> IO (Either BigExc a)
getFooBarIO = undefined

bigToSmall :: BigExc -> Maybe SmallExc
bigToSmall = undefined

runFooAPI :: Members '[Embed IO, Error BigExc] r => InterpreterFor FooAPI r
runFooAPI = interpret $ \(GetFooBar str) -> fromEitherM (getFooBarIO str)

runFooAPIExceptional
  :: Members '[Embed IO, Error BigExc] r
  => InterpreterFor (Exceptional SmallExc FooAPI) r
runFooAPIExceptional =
   runFooAPI
 . runExceptional bigToSmall
 . raiseUnder

The key combinator of interest here is exceptionally, as well as the interpreter runExceptional.
This has the benefit of shuffling more responsibility from the user to predefined functions. You don't need to understand how it works to use it!

Yeah, it uses a lot of internal stuff, but that's fine. We'll probably add Exceptional, or some generalization thereof.
Also fun fact the only reason Exceptional instantiates its parameter is because weave has a constraint it doesn't need. We'll fix that.

nice, I'll have to try that out in the wild to assess the ergonomics.
For the record, this is what I bruteforced together to achieve the UX I was going for:

data E = E deriving Show

data A :: Effect where
  DoA :: A m Int

makeSem ''A

data InterpreterError a err =
  InterpreterError err
  deriving Show

data EffectError eff err :: Effect where
  InterpreterCatch :: m a -> (err -> m a) -> EffectError eff err m a

makeSem ''EffectError

interpretEffectError ::
  ∀ e err r .
  Show err =>
  Member (Error (InterpreterError e err)) r =>
  InterpreterFor (EffectError e err) r
interpretEffectError =
  interpretH $ \case
    InterpreterCatch ma recover ->
      (catch (run ma) (\ (InterpreterError er) -> run (recover er)))
      where
        run =
          raise . interpretEffectError <=< runT

interpreterError ::
  ∀ eff err err1 r .
  Show err =>
  Member (Error (InterpreterError eff err1)) r =>
  (err -> err1) ->
  InterpreterFor (Error err) r
interpreterError lift =
  interpretH $ \case
    Throw er -> do
      throw (InterpreterError (lift er))
    Catch ma _ -> do
      raise . (interpreterError lift) =<< runT ma

observeEffectError ::
  ∀ e err err1 r a .
  Show err =>
  Show err1 =>
  (err -> err1) ->
  Sem (Error err : EffectError e err1 : Error (InterpreterError e err1) : r) a ->
  Sem r a
observeEffectError lift =
  eek <=< runError . interpretEffectError . interpreterError lift
  where
    eek (Right a) = pure a
    eek (Left _) = pure undefined

run ::
  Member (EffectError A Text) r =>
  Member A r =>
  Sem r Int
run =
  interpreterCatch doA $ \ _ -> pure 1

interpretA ::
  Member (Error E) r =>
  InterpreterFor A r
interpretA =
  interpret $ \ DoA -> throw E

test_effectError :: IO ()
test_effectError =
  print =<< (runFinal . observeEffectError show . interpretA $ run)

It's got a few holes :slight_smile: