freenode/#lisp - IRC Chatlog
Search
22:02:50
aeth
_death: I've settled on something like this as a good compromise between having to know/hardcode too much about HTML (really, you only need to know the empty elements) and being easy to write. https://gitlab.com/mbabich/cl-documents/blob/6d6cd886aa1ff64c349a72f0957943e431489ab9/generate-html.lisp#L175-194
22:03:33
aeth
_death: The only remaining issue is that it is much more complicated to do this dynamically than statically, i.e. replacing (:p "Hello, world!") with (:p "Hello, " person "!")
22:06:01
aeth
That's because there's now a layer of indirection. You'd need to write the HTML markup to a string, and then write that string to the desired stream at run time. Technically, it'd probably be faster to do that even statically.
22:08:41
TMA
aeth: nay. for <a><href>https://example.com/</href>example you would have (a () (href () "https://example.com/") "example") and that would be unambiguous
22:09:18
aeth
TMA: Yes, if you want to always have a () that is an alternative possibility, I didn't consider that
22:09:28
minion
The URL https://gitlab.common-lisp.net/users/sign_in?secret=88a32f2a will be valid until 22:15 UTC.
22:11:30
aeth
It's more verbose for "plain" HTML, but there's certainly a lot of generated HTML that is full of (span :class foo ...) that could just as easily be (span (:class foo) ...)
22:15:31
_death
aeth: yes, that's what good html generators do.. they interleave the static HTML fragment strings with the code to generate the dynamic parts
22:17:09
_death
I've not written any web stuff these recent years, but yaclml would still be my go-to library for this
22:32:44
_death
it does differentiate between tags and attributes.. as you come up with your own tag macros you can choose to do away with the latter I guess, though in my projects I did not
23:01:51
Josh_2
I have a question about how you are writing zs3. I am just curious, I was reading through some of the sourcecode and I noticed that you are generating lots of lambdas based on the value of 'form' (I think) why are you writing the code like this instead of using clos and generic functions?
23:02:22
aeth
_death: It would probably be wiser to use a library but (1) most libraries I've seen don't solve the problem in a satisfactory way, in that you're mostly just format niling a bunch of intermediate strings around the place at runtime and (2) I'm kind of going for a more, idk, pandoc approach?
23:03:09
aeth
I think I'm going to try org after markdown, although if I do that I'll probably never finish
23:09:35
_death
aeth: heh, I guess it's another rite-of-passage thing.. to write your own html generator
23:23:36
aeth
_death: Well, generating HTML is kind of secondary to my goal here, although not secondary enough for me to rely on a library to do what I want.
23:29:52
aeth
_death: I mean, exactly yes, it's just that the documentation is incredibly elaborate and difficult compared to an ordinary library because the library I am going to be documenting is a game engine
23:31:08
aeth
_death: So I need to essentially be able to document linear algebra, computational physics, computer graphics, etc. And with an insane target like HTML (or XHTML) instead of a sane target like LaTeX
23:31:54
aeth
_death: Hence the several month diversion from the engine itself (although the commits only go back a few weeks)
23:39:01
aeth
_death: I'd love to be able to explore different routes, but I'm kind of forced into one of two routes (not mutually exclusive): in-browser documentation and in-editor documentation... assuming my engine will one day get an editor like most game engines tend to bloat into having these days... except I don't have the latter at this point.
23:41:05
aeth
_death: texinfo->html might work, but I suspect the subset is far too much of a subset for my needs. If I had to use an external markup+tool, I'd probably try to do some LaTeX->HTML
23:42:03
_death
aeth: yeah, I guess markdown is reasonable for that.. so when you have an editor you can easily render it
23:47:50
aeth
_death: The important characteristics about Markdown are (1) you can embed HTML into it to get features it doesn't support and (2) it's commonplace to extend it (especially tables) to fit a particular domain. Also, familiarity, I guess, but I personally have used org, Markdown, restructuredtext, MediaWiki markup, BBCode, LaTeX, etc., and they all have advantages and disadvantages and it's not hard to pick up quite a few of them.
23:49:40
aeth
The problem with org, btw, is that you don't implement org, you "export" a subset of org from its native GNU Emacs environment... e.g. you're not going to be supporting https://orgmode.org/manual/The-Spreadsheet.html#The-Spreadsheet
23:52:15
aeth
_death: Of course, the disadvantage of Markdown is that if you do use that HTML feature, it's not as simple to implement a new, non-browser backend for it since the language then isn't as simple as it seems.
23:52:39
_death
yeah.. some years ago I used org for writing documentation and generated pdf/html.. I used emacs in batch mode, but also had to hack htmlize.el a bit to get what I wanted
23:53:33
aeth
yeah, that's the thing... you kind of need to control the whole pipline if you want (a) input->HTML to work and (b) input->something-else to eventually work using the subset of features that you need it to support
23:55:00
_death
hmm, either htmlize or org-export-as-html or both.. apparently I hacked both for some purpose in the past :)
0:00:02
aeth
there really isn't way to do this that makes everyone happy... unless you heavily CLOSify things so that they can effectively rewrite large parts of what you're doing via :around methods
0:01:45
_death
keep the markup language constrained.. don't allow html escape, but do allow more limited extension
0:02:57
aeth
_death: What I'm going to do is I'm going to actually parse the HTML in the Markdown files so that I can eventually translate *those* into my intermediate format(s) as well as the Markdown itself.
0:03:22
aeth
So... now I have to write a subset of an HTML parser. I'm going to probably be stricter than some who lazily pass through the HTML or something to let browsers magic together a best-guess of what you meant
0:03:58
aeth
But now it suddenly turns one of the simplest markups into also including (a subset of) one of the hardest, so I'm going to procrastinate that for a while
0:05:20
_death
what about mathematical formulas.. it's not hard to extend markdown to support, say, mathjax syntax.. but now you need to write your own mathjax
0:11:43
aeth
_death: Well, presentation MathML should trivially follow from HTML generation. I mean, it's hideously ugly, but that only will affect generation speed a bit and memory a bit, since you never have to directly touch it. https://en.wikipedia.org/wiki/MathML#Presentation_and_semantics
0:12:49
aeth
I also already deal with changing the writer in the middle of writing the content (for directly-embedded CSS in <style>) so that's just going to be yet another special case tag
0:16:00
aeth
_death: I thought MathML is supported in modern browsers and something like MathJAX is for legacy browsers? Although most people who generate equations rarely check that the browser can actually do it without having to go down the legacy branch (another alternative many do, e.g. Wikipedia, is generating images)
0:16:24
aeth
_death: but, yes, there would also have to be another path for the native GUI, when that happens
0:18:53
_death
aeth: I dunno.. some time ago I wrote a static blog generator with a friend that rendered markdown using 3bmd.. it had support for writing latex to be rendered by mathjax, since the friend was used to writing in that notation
0:19:57
aeth
_death: Oh, weird, Chrome removed it. https://en.wikipedia.org/wiki/MathML#Browser_support
0:20:32
aeth
_death: Yet another case of a monopolist holding technology and standard back, I guess. Their solution "use JavaScript" is, of course, good for their bottom line since using JavaScript enables the tracking that drives their megacorporation.
0:20:51
aeth
I think I'll use MathML and MathML alone anyway. Maybe with a link to download Firefox. :-)
0:21:43
aeth
Basically if you're just a Chromium reskin, you can't use MathML, but if you're any other browser (even Webkit) you can. https://caniuse.com/#search=mathml
0:22:51
aeth
(okay, IE/Edge don't support it either, but IE supports nothing interesting and Edge was basically trying to be functionally compatible with Chromium until they decided to eventually switch their backend to Chromium to save the effort)
0:40:24
_death
since browsers are gluttons for complexity, why have mathml instead of the notation that everyone uses already (latex)
0:41:28
_death
it would make my life easier, for sure.. (I use a browser with js disabled by default.. every site that uses mathjax w/ latex I need to consider and perhaps reopen with a js-enabled browser)
0:41:39
aeth
_death: the general idea is that you would parse the LaTeX subset that everyone uses for equations into (something like) mathml and never write it directly
0:42:07
_death
also, since you talk about javascript that way... it's surprising that you use gitlab, another annoying site that requires javascript to display source code
0:42:16
aeth
supporting (the subset of) math LaTeX would set the precedent to do things like directly supporting Markdown
0:43:47
aeth
_death: Ah, weird, I thought it only used JS for the syntax highlighting and not the source code entirely, which is fairly common. Requiring JS for a web application is very different than requiring it for a document, though.
0:43:56
_death
yes.. at this point nobody in their right mind contemplates writing a browser.. so adding these things is no big deal
0:46:07
aeth
_death: Imo, JS in an ordinary web page should only be used for features that aren't essential... so requiring it to render math equations is absolutely unacceptable, since it's essential to the nature of the page (displaying the non-application document correctly). Doing the Wikipedia approach and making equations images is complicated and anti-accessibility (and also makes it hard to search the page)
0:46:38
aeth
(Even though it is a hideous monstrosity to look at directly and should never be written by hand)
0:48:07
aeth
_death: Gitlab requiring JS to display source code actually *is* pretty bad, and it's gitlab-static.net so it requires a whitelist of JS to get it to load in umatrix (which only allows first-party JS by default)...
1:00:03
aeth
I mean, technically, it could just add rendering to the HTML parser that I'll have to write to write a complete Markdown parser, when I get to it... but you kind of want compatible rendering, not just rendering.
1:04:48
vms14
I know nginx is good for a load balancer, and it seems to be a very convenient way to use it as a proxy, so it could serve static stuff and use hunch for dynamic stuff
1:12:55
vms14
knowing I have no thread support in sbcl because I use netbsd for me seems even a better option
1:13:44
aeth
vms14: While technically I am going to have to do "dynamic" HTML as in not-at-macro-expansion-time HTML (for the markdown), I'm writing static HTML here, for Gitlab Pages.
1:22:20
aeth
vms14: parsing markdown, writing HTML. Technically, the way I write my parser (which I consider idiomatic for CL) makes it trivial to turn into a reader macro since it uses streams
1:35:30
_death
aeth: so, here's my hack for the day: https://gist.github.com/death/12141afa3b3252e991003bc18cb5760d ;)
1:37:09
_death
no-defun-allowed: it's a greasemonkey script to fetch and "transclude" the source code without needing to enable js in firefox
3:39:13
aeth
seok: The performance is probably going to be worse, but only because of lack of attention, not because of lack of potential. In CL, you can use native arrays for high-performance code, while in Python you have to FFI out to Fortran or C or C++ using something like Numpy.
3:40:12
aeth
seok: except you can basically do the same thing and, for 1D arrays, use FFI with them and call out to the same optimized Fortran or whatever that Numpy uses in its backend
3:40:45
aeth
But unlike with Python, there's nothing stopping you from writing Fortran-competing numerical code in CL afaik... it's just, a lot of work.
3:42:32
aeth
seok: I'm guessing that natively in CL it's probably going to be about 3x slower than the equivalent Fortran/C/C++ because the compiler won't SIMD your operations and there probably aren't many libraries to do that for you with inline asm for SBCL.
3:42:54
aeth
seok: Maybe not as bad vs. Numpy because the FFI is going to add some overhead back in.
3:43:58
aeth
SBCL is basically the only implementation you're going to have a good time writing numerical code in, even though it will technically run everywhere (as long as you have a slow path if you write a SBCL-specific fast path in asm)
3:44:17
aeth
SBCL does more optimziations for this kind of thing, so slower compile time but faster run time.
3:46:20
aeth
But if you care about performance, you're probably going to have to FFI not because you need to FFI to write fast numerical code but because there isn't much numerical code in native CL.
3:47:04
aeth
Technically that's not quite true. Maxima, a CAS, compiles Fortran to CL, but it uses this antiquated library that generates garbage numerical SBCL. Something written with SBCL's optimizations in mind could probably produce something that generates considerably faster asm when you run DISASSEMBLE.
3:49:09
seok
Just thought it would've been cleaner if I could do it in CL if I don't have to give up performance
3:49:58
aeth
Are you willing to write those routines yourself? If you are, you absolutely don't have to give up performance.
3:50:37
aeth
The tl;dr of optimizing SBCL is that you want to do everything in structs of arrays of specific numerical types, ideally with a known length at compile time.
3:51:38
aeth
If you're working with integers that are larger (or smaller) than the fixnum size but that can fit in signed-byte or unsigned-byte 64, or if you're working with double floats, you want to keep them in arrays and work with mutations so that you don't heap allocate. (optimize (speed 3)) will warn you in SBCL
3:53:32
aeth
seok: you want to keep your data in arrays with an :element-type that the implementation can optimize. If you have heterogeneous data, you want to do the struct-of-arrays pattern and have multiple parallel arrays in one struct, since the arrays must be homogeneous.
3:53:46
aeth
If you're using single-float it's a bit easier because you don't have to worry about accidental heap allocations like with double-float
3:55:00
aeth
seok: Now you have a (simple-array single-float (42)) and you have to add type declarations (type hints) so that the compiler can optimize things more fully (although idk if any do this other than SBCL)
3:56:49
aeth
seok: simple arrays of an element-type that is not T, or those structs with slots with a :type provided, ideally with a known length so bounds checks can be removed
3:57:01
aeth
e.g. this is very optimized: (defun foo (x) (declare (type (simple-array single-float (42)) x)) (* 2f0 (aref x 37)))
3:57:29
seok
So, I don't know how numpy works in the back, for matrix operations, will optimized loops in CL be just as fast?
3:57:33
aeth
If you use double-floats, things get harder... that example but with double-floats wouldn't work well because it would heap allocate the double-float result. With doubles, you have to mutate more if you want to avoid this. Or inline functions.
3:58:11
seok
Well, data provided is only given to 6 decimal points anyway, don't think doubles are needed
3:58:37
aeth
seok: Modern processors have asm specifically intended for linear algebra, SIMD... single instructions multiple data. So... The numpy routines probably do heavy compiler hinting or inline SIMD to get this.
3:59:19
aeth
SBCL has some SIMD support but it doesn't do it automatically, so you're going to be working one value at a time.
4:00:39
aeth
If matrix performance is an issue, you could call into a BLAS library for matrix multiplication using something like https://github.com/sionescu/static-vectors/ to give you vectors that are both CL-native and C-native, easy to use in FFI
4:00:42
seok
I haven't noticed any multi thread activities when I was working with numpy some time ago
4:01:09
aeth
static-vector for matrices has two disadvantages: (1) you have to free your data and (2) you have to use 1D arrays, while matrices are more idiomatically expressed with 2D arrays in CL
4:01:38
aeth
(the whole "static" thing is so that the GC doesn't move the data around so C code can use it)
4:03:46
aeth
It's very easy to e.g. have an API that copies the vectors and destroys performance, and static-vectors is probably too new to be a dependency for many of those
4:04:11
aeth
It's absolutely possible to do efficient linear algebra in CL, both natively and FFI, though.
4:04:52
aeth
Just doing it natively in CL but with many threads might beat Python+Numpy, especially if you're using a Ryzen 9 3900X or a Threadripper or similar processors.
4:05:27
aeth
If Python+Numpy is single threaded and 3x faster, I'm guessing if you throw 24+ threads (R9 3900X) at the problem, it goes away.
4:08:52
aeth
seok: Oh, and never assume that single-floats are sufficient, btw, unless you're using numerical algorithms that tell you that.
4:10:32
aeth
The main thing about double-floats is that you never want to return an individual double-float or call a non-inline function with an individual double-float (by "individual", I mean not in a struct or an array)
4:11:32
aeth
As long as they're in some other box (or if they stay within the scope of one function), they won't be boxed, and that greatly improves performance.
4:14:22
aeth
It's just a side effect of how most CL implementations work: something is heap allocated unless it can (with its type tag!) fit in a native word. So a 64-bit type can't fit because it needs a tag.
4:17:39
aeth
Although if you have so many things, maybe there's a memory constraint to use singles...
7:31:01
MichaelRaskin
aeth: re: Wikipedia approach: Images with LaTeX source in alt= is not _that_ bad, is there a good accessibility solution that is actually better for MathML?