Data from DB? - Rib

Is it possible to wire-up Rib in a way such that the data for the static site from from Postgres tables? How would the build process figure out what has changed, and what hasn't? Will it end-up re-building the entire site every time?

@Saurabh Nanda I saw your question on the Slick tracker, and was wondering what you were planning to build. Could you explain the original goal?

Are you for example trying to create static blog but with comments backed on postgres, and you want to render the comments as well statically?

Speeding up a traditional CMS-powered website by pre-generating pages that change infrequently.

Both rib and slick use Shake -- in particular Shake.Forward -- as its build engine. So your question is more geared towards Shake itself I think.

Shake is conceptually like Makefile, but defined as a DSL in Haskell.

When using rib, you normally call Shake's readFile' which automatically makes that file as a dependency for the current target, so when that file changes, the target is rebuilt.

You would probably build a combinator like that, but using database instead of the filesystem as the source.

Maybe one could do something really interesting using postgres's LISTEN/ NOTIFY mechanism?

hmm interesting. looked at rattle, which is probably the inspiration for Shake.Forward

would you know how something like Shake.Forward works if it reads a file listing from a directory (as input), and then the new files are added to the directory? will it be able to detect the change, and re-run the build commands?

and isn't that something that rib has to contend with?

Shake doesn't "monitor" filesystem changes like that. Which is why rib uses fsnotify.

re: rattle - interesting! never come across it before; it is from the same author, and is being actively developed.

so IIUC, if your build process reads 3 known files, then Shake.Forward will automagically mark them as deps. But if after that a 4th file is added as a dep, shake won't rebuild?

rib uses fsnotify directly, or via ghcid?

Right, as long as you use something like cacheActionWith in your "forward" rules.

rib uses fsnotify directly; but ghcid also helps with the scenario of Haskell sources changing (which should do a rebuild, because HTML/CSS in the Haskell DSL may have changed)

Sridhar Ratnakumar said:

Right, as long as you use something like cacheActionWith in your "forward" rules.

didn't understand this statement. what will happen as long as you use cacheActionWith?

this "But if after that a 4th file is added as a dep, shake won't rebuild?"

cacheActionWith prevents those rebuilds.

so, shake.forward will normally figure out a newly added dependency, or one is forced to use cacheActionWith to teach shake how to figure out dependencies? And if the case is latter, then doesn't it become a backward build system all over again?

Sridhar Ratnakumar said:

cacheActionWith prevents those rebuilds.

but I'm expecting to shake to rebuild, but can't figure out how shake would come to know of a 4th new dependency.

say, for example, a YAML file contains a list of 3 markdown files that need to be compiled. when the build command is run, Shake.Forward will track one YAML file + 3 files that are read after reading the YAML file, as deps, right?

how does shake know that the 3 files that are being read, are a result of the contents of the first YAML file?

If you use Shake's getDirectoryFiles - then it tracks them. Otherwise you will have to call the [need] function manually, which explicitly tells shake to track the argument as dep.

That's what the Dhall parser does (call need explicitly): https://github.com/srid/rib/blob/master/src/Rib/Parser/Dhall.hs#L33

which basically means that shake.forward is very brittle, (explicity marked as experimental in the docs)

Not the ideal build system, yea.

is shake.forward core to rib ?

Just taking a step back, and thinking about this problem in a more general fashion: I'd like to explore the FRP model for tracking changes and building 'final values' like we do in a static site.

I was thinking of FRP, but stumbled across slim and rib and wanted to evaluate this space first.

Saurabh Nanda: is shake.forward core to rib ?

yes

and shake.forward is not going to work its magic with the DB obviously

Is this a projet you have taken to work against an existing CMS? Or an in-house system? Or do you have the flexibility to design this system from the ground up? Who are going to be its target users?

have the flexibility to design ground-up

The Postgres LISTEN/NOTIFY mechanism is used in obelisk apps to get postgres notifications to the frontend's FRP network. Basically what I do in my app http://www.cerveau.app/ (which is the web app version of neuron)

what happens if a notification is missed?

it's part of rhyolite that nobody outside of obsidian really uses haha, but it is pretty cool tech: https://www.srid.ca/2012401.html

well, if you refresh the page, it is going to query the initial data anyway (which does a normal sql query), and from that point in your "single page app" data will come in automatically (no need to refresh).

consider a server side static-website system. what happens if the template itself changes, and the site needs to be re-generated? will the ENTIRE site need to be re-generated?

have the flexibility to design ground-up

Why a statically generated site? Instead of a SPA one like https://reflex-frp.org/ ?

(a) the content is mostly static, (b) SEO, (c) Google PageSpeed

consider a server side static-website system. what happens if the template itself changes, and the site needs to be re-generated? will the ENTIRE site need to be re-generated?

If template changes, yes. What's wrong with that?

as in, site uses multiple templates, and one template changes which probably affect 10% of the pages, will the entire site be rebuilt?

SEO is important. This zulip's archive is over at https://funprog.srid.ca/ - it is generated using rib, but the "input data" is not local files; rather dynamically read from zulip API (though an intermediary json file is used to trigger Shake)

as in, site uses multiple templates, and one template changes which probably affect 10% of the pages, will the entire site be rebuilt?

Only those affected should be rebuilt.

If there is no obvious dependency link, you'll have to render them anyway to be able to tell what those 10% are (just not write the 90% to disk)

wrt rib what would be the best way to pull data from the DB?

You would only be using the nix/ghcid part of rib in that case (not shake/fsnotify). And do all the plumbing yourself. Unless you do a trick like funprog.srid.ca does (i.e., write intermediary files)

so, getting data to a file is necessary for the magic to start

that's one approach, yes. another is to design from scratch, preferably using a FRP model? the latter might be too research-y.

let me keep abreast of your findings either way! i'm interested in this problem as well.

everytime I think about FRP, I can't figure out how it would work if the state is sitting outside the FRP system (the DB, in this case)

it is hard problem for sure. obsidian will have something cool to show later this year, or after that.

(but rhyolite works meanwhile, even if a bit cludgy)

https://materialize.io/ is half the equation for such FRP app

listen/notify is one technique, and assuming you listen to all changes being made to the DB, how do you select the changes that affect your current FRP network?

in rhyolite, there is a type called "ViewSelector" which allows the frontend to select all it needs from various points in network. Globally they are combined (Monoid), and communicated via websocket channel.

you don't listen to all changes. and you 'notify' explicitly, from the points in your backend where you update. and then there is a NotifyHandler that handles the notifications, and updates the resulting "View" (based on what's selected in ViewSelector) which gets merged downstream

this app does all i've said https://github.com/srid/obelisk-rhyolite-template

by the way, in case if you are wondering: reflex is a pretty isolated ecosystem in the Haskell universe. you won't get help for it in places like FP slack. IRC (#reflex-frp) is the only place where prominent folks can respond to your question. they are very helpful.

(and I recommend riot.im for IRC; it keeps everything persistent! or irccloud )

cool, thanks.

let me read-up some more and think some more.

the core reflex engine is not tied to web apps. it can be used in non-web contexts too, eg: https://github.com/reflex-frp/reflex-vty