freenode/#lisp - IRC Chatlog
Search
16:52:57
jfrancis
I have a bunch of code that reads and processes files produced by a third party (the FAA, specifically). About a week ago, they started adding a BOM (unicode byte order mark, 0xef 0xbb 0xbf) to the front of every file. I'm pulling my hair out, tried everything I can possibly think of to skip past those first three bytes and read the rest of the file (which is just plain old ASCII). No luck. Anyone dealt with this? Any thoughts? 64-bit SBCL 1.4.5.38-9ee5e08
16:56:04
jfrancis
Perhaps, but if I can, I haven't figured out how. Adding ":foreign-encoding :utf-8" to with-open-file didn't hack it. Although at least the errors changed.
17:04:56
jfrancis
Oops, obviously, I mean :external-format, not :foreign-encoding. Confusing my languages. Using either :latin-1 or :iso8859-1 gives me the error "The bounding indices 40 and 45 are bad for a sequence of length 42."
17:05:46
jfrancis
I could strip the first three characters from each file as a pre-processing step before I feed them to my code, but that's precisely what I'm trying to avoid.
17:06:42
jfrancis
Or I could send a letter to the FAA and ask them to go back to publishing the files the way they've been doing it for ten years up until this Monday. But I doubt that will get much traction.
17:30:44
_death
jfrancis: what happens if you just (read-char stream) after opening the file to skip the BOM
17:43:38
_death
and the sbcl manual mentions a :replacement option that can be supplied to external-format, so that may also work
17:49:19
jfrancis
Sigh. It just seems horribly broken that I can't read a fairly common (though horribly broken) file format without ridiculous gyrations. Sticking a Byte-Order-Marker on the front of a file, despite the inherent evil of it, is common.
17:51:48
ZombieChicken
jfrancis: If it breaks a standard or you can explain how it's broken, then maybe the email makes more sense. Perhaps whoever made that change was just ignorant
17:53:49
jfrancis
ZombieChicken: Here's the problem boiled down to it's essence: How can I read an ASCII file in a portable way in CL that has a BOM on the front of it. Assuming, of course, that I can't mercilessly flog the person who put the BOM there in the first place.
17:55:06
ZombieChicken
jfrancis: I'm aware. I'm also not well acquanted with CL, so I can't offer any real help
17:56:39
_death
if it has a BOM, it's not ASCII.. and if you want to do it portably, you need to use a binary stream and do the decoding yourself
17:57:19
ZombieChicken
But if that appended data is nonstandard, an email and a prayer you get to someone who isn't stupid would be the best option; working around and excepting a broken file format doesn't help anyone
17:58:36
ZombieChicken
_death: Possible to grab a binary stream, read a few bytes, then convert the rest to an ASCII stream?
17:58:37
jfrancis
ZombieChicken: You're not seriously thinking an email to a US Gov't Federal Agency will get them to change a file format, are you? :D
17:58:49
pfdietz
Does sbcl provide restarts for that reader error? Maybe ask for one that would help here.
17:59:29
ZombieChicken
jfrancis: No, I don't expect it to do anything useful unless you're bribing people, but you might get lucky
17:59:35
jfrancis
Yeah, that's what I've resorted to. Opening the file as binary, discarding the first three bytes, then reading in the rest and converting it back to a string. It's unspeakably inefficient. Then again, it only ever has to be done once per file.
18:00:57
_death
ZombieChicken: yes, you can do precisely that with a bivalent stream.. in sbcl you can open a bivalent stream by providing :element-type :default
18:02:38
jfrancis
So I can open a stream, do three (read-byte ), then read the rest as strings? If so, is that portable, or is that an SBCL-ism?
18:04:21
jfrancis
Hmm. Ok. I just got my other code working (that reads it as binary, then converts the byte vector to ASCII). But that code is so embarrassingly bad, I'll go look at flexi-streams. Thanks!
18:15:18
jfrancis
Heh, it runs out of heap when loading the 182meg input file. Guess I need to go figure out the "right" way.
18:28:19
pierpal
182MB is very little data, my toaster has more than that ram. How are you reading those files?
18:40:29
antoszka
shka_: That talk is really depressing. I've seen it a few years ago, it shows how we've been pretty much running circles since the 60's.
19:21:15
cgay
Claiming that people are *trying to kill lisp* seems a little self-important to me. :)
20:34:47
makomo
robert smith did a video on lisp with computerphile: https://www.youtube.com/watch?v=svmPz5oxMlI https://www.youtube.com/watch?v=dw-y3vNDRWk
20:36:18
aeth
Having spent a *LOT* of time with Common Lisp, there's nothing magical or special about it.
20:36:35
aeth
What it has is the lack of a misfeature that virtually every other programming language has: elaborate syntax.
20:37:44
makomo
in the second part, i think robert mentioned EVAL way too many times, and gave everyone the impression that somehow it's about EVAL
20:40:12
aeth
There are three advantages to the simple syntax. Lisp gives your source code a level of polish you literally cannot get anywhere else (okay, there's probably a few other places) without writing your own programming language. And doing things at compile time is just about as easy as doing things at runtime, which is rare. (Similar levels of polish to point #1 are probably at a runtime cost, not at compile time.)
20:40:45
aeth
And you don't have to revise its standard every 4 years to give something new a convenient form in your language's syntax.
20:42:26
aeth
I don't think Lisp popularizers generally hit on my points because... they're not magical.
20:42:51
makomo
aeth: well i agree, but that sort of thing comes after a certain amount of time/experience, and i don't think it applies just to lisp
20:44:10
pjb
makomo: https://dspace.library.colostate.edu/bitstream/handle/10976/166750/Kilaru_uccs_0892N_10293.pdf?sequence=1
20:45:01
pjb
makomo: with sexps, code injection are not possible. with text, they're common occurences.
20:45:17
makomo
aeth: in general i think experienced people tend to forget the little things that bothered them when they were just starting out
20:45:40
makomo
and then instead of emphasising certain stuff (that they found confusing early on), they just gloss over it or assume it should be clear to everyone
20:45:58
pjb
makomo: lisp invented in 1959, would prevent thousands of students getting PhDs, tens or hundreds of thousands of companies having contracts to "prevent" or correct the problem, that wouldn't exist in the first place.
20:46:36
makomo
pjb: i've already been sold on lisp :-). i'm mainly referring to the comments on the video
20:47:03
pjb
makomo: it's the same thing with OSes. Without unix and C, (ie. if you used controlled execution programming languages such as lisp (without FFI), and capability-based OSes, you wouldn't have jobs for millions of programmers and computer security engineers.
20:47:41
pjb
makomo: basically, you'd halfe the computing GDP, and you'd have to make all those people work on solar system and galaxy colonization instead…
20:47:46
makomo
pjb: sometimes i wonder where would lisp be today if all of the effort that went into other languages went into *just* lisp
20:48:15
ZombieChicken
pjb: or they would be working on more useful and interesting problems than trying to (re)fix problems
20:48:21
aeth
makomo: Imo, Lisp was hard for me to learn initially because all of the material was treating it as this special magical thing (which is probably more accurate for Haskell than for Lisp). I picked it up at a much more rapid pace once I realized it was basically like any other language, with some key style differences, like relying on return values from everything being an expression rather than on setting variables. (e.g. use a cond to retu
20:48:52
aeth
(e.g. use a cond to return one of several possible values, not an if/then/else block to set a variable)
20:49:56
makomo
aeth: mhm. i feel like that's a pretty common thing. reminds me of monads and burritos a little bit :-)
20:56:29
nirved
emaczen: and better explanation here https://groups.google.com/forum/#!topic/sbcl-help-archive/6vvc1qIc8vU
21:35:35
makomo
assuming a compiling lisp implementation (is it even possible for a cl implementation to not be compiling, since stuff like COMPILE and COMPILE-FILE are required?), what is the difference between lisp's "on the fly" compilation and JIT compilation?
21:36:22
makomo
is JIT just a subset of "on the fly" compilation? i.e. the moment when the code is compiled is (drum roll) just before running the code? while in lisp i can compile whenever i want?
21:37:30
White_Flame
(in most circumstances; having a separate COMPILE call might do different things, but in practice, defining the function puts it in its final state that the implementation supports)
21:38:26
White_Flame
most JIT systems look at collected heuristics durign runtime, then recompiles that same original source with different optimization assumptions
21:38:58
White_Flame
so it can generate native machine code multiple times for the same source code, as it learns more
21:39:53
White_Flame
deciding which calls to inline is also a decision that can be made after first run, depending on the sorts of call overhead it's measured
21:40:10
makomo
i see, but none of that is inherent to "*just* in time" compilation right? i.e. couldn't you do the same thing with ahead-of-time compilation?
21:40:35
makomo
i mean, you learn something about the code, and then recompile parts of the code, but not *just* in time, rather *ahead* of time, just to be run later
21:41:20
makomo
so let's say you're not defining a function but evaluating a form like (+ 1 2) at a lisp REPL
21:41:23
White_Flame
"just in time" can also imply the time that it's needed to be faster, as you notice where the timne is being spent
21:42:11
makomo
so then really the key point about lisp's compiler is its availability at runtime and the interface for the user
21:42:37
makomo
i.e. it's not part of some runtime system you can't touch/see, but you can actually invoke it yourself, as the user
21:42:51
White_Flame
so if you have something declared inline, or redefine a macro, those effects won't get recompiled into prior-compiled functions
21:45:28
makomo
would you know how usual lisp compiling implementations work? i mean i get that generating machine code and executing it at run-time is not really any different than producing an executable and then executing that (ala C/C++/etc., but still)
21:46:01
makomo
so when something gets compiled, the machine code gets placed somewhere in memory and the control is transfered to that machine code whenever the function is invoked for example?
21:48:23
makomo
that's pretty cool, because that sort of "runtime code generation" is something you get to see while reverse engineering malware or self-modifying code, etc.
22:27:37
aeth
e.g. if a type is observed through some runtime to be common you could replace some running-in-Lisp language's function with a function that has a typecasw where the first argument is a faster function written for that type
22:34:50
stylewarning
Here was round 2 of Computerphile and Lisp. Pardon the few gaffes you Lisp experts! https://youtu.be/dw-y3vNDRWk
22:38:04
stylewarning
makomo: thanks for the feedback. I think you raise a lot of great points. It seems people got stuck on EVAL, which is a necessary component of the process, but maybe should have been buried a bit deeper. But if I said “macro”, people would likewise get up in arms.
22:39:01
stylewarning
In the numerous times I’ve tried to explain these concepts to people in less than 10 minutes, I’ve come to the conclusion that either the concept is somewhat intrinsically difficult to install in your brain, or I’m not a very good teacher. :)
22:40:13
makomo
yeah, i think a lot of people failed to see past the "woooow, what's so great about EVAL"
22:40:58
stylewarning
The prompt was: “explain something well enough so a videographer could learn at least one new thing” :)
22:43:14
makomo
one of the comments brought up something like "how is this different from C's (or C#'s, i'm not sure?) #define", which is an okay question imo
22:43:34
stylewarning
The first take of both videos was a bottom up description of the syntax of an S-expression followed by the semantics of EVAL, a la Allen
22:49:52
drunk_foxx[m]
There is a preview at the end of the video, tends to be about quantum computing
22:50:37
stylewarning
I also wrote a tutorial on how to write a quantum interpreter in Lisp that I’d love feedback on before wider publication
22:55:16
makomo
i wish that your rigetti talk had better audio, but i did watch it anyway, although it wasn't very pleasant :-)
22:57:02
stylewarning
makomo: drunk_foxx[m]: here’s the DRAFT of the tutorial. Feedback to robert@stylewarning.com would be appreciated! https://www.dropbox.com/s/n5k9pikkcx5besa/QUANTUM_INTERPRETER_DRAFT_STATUS.pdf?dl=0
22:58:42
makomo
stylewarning: a quick question, is your old blog done for good? i.e. the only way to access is through archive.org?
23:02:14
stylewarning
It’s done for good. It’s been on the back burner to get that content back up on stylewarning, but the db was hosed, so I have to copy manually
23:05:59
stylewarning
aeth: but it’s considered contrived to many because everybody already knows what a for-loop is (:
23:06:33
aeth
But I think many programmers can read between the lines and think about what loop construct they'd like to write as a macro.
23:07:21
aeth
e.g. It'd be trivial to write a foreach on top of map nil or loop, even though CL doesn't actually have one.
23:07:26
makomo
stylewarning: yeah, that's what i read out of the comments too. people are so accustomed to for loops that they take them for granted, but i do hope they could see the point
23:08:52
aeth
e.g. (foreach (foo foos) (whatever foo)) could just be macroexpanded into (map nil (lambda (foo) (whatever foo)) foos)
23:12:47
aeth
for (ending with oh, hey, it's in the language as dotimes) -> foreach (trivial on top of map nil) -> something more elaborate, maybe
23:13:50
stylewarning
Interviewed while I was in England to do a whole different thing (explain how to program a quantum computer)
23:14:25
aeth
Well, you could always start a YouTube channel (if you don't already have one) and take advantage of the YouTube related algorithm that would hopefully pick up a more in depth (20-30 minute?) video
23:17:01
aeth
(And in case anyone didn't pick it up, the difference between a foreach implemented via map nil and a foreach implemented via a portable usage of loop is that map nil is generic to all sequences, and the loop-based foreach would have to choose between lists or vectors.)
0:05:13
emaczen
Then I started SBCL and I get a TYPE-ERROR "QL-UTIL" is not of type (OR FUNCTION SYMBOL) in quicklisp/setup.lisp
0:06:02
Bike
but if you built sbcl wrong that usually manifests as more like, lacking an extension, or just outrifhg failure
0:18:20
jfrancis
In case anyone is still around from when I asked this morning about how to strip the Unicode Byte-Order-Mark from the beginning of an otherwise-clean 8-bit ASCII file, the suggestion of flexi-streams worked wonderfully. And is even portable. Thank you.
0:38:46
emaczen
why does SBCL keep failing to load quicklisp.lisp? I get the error: QL-UTIL is not of type (OR FUNCTION SYMBOL)?
0:47:13
emaczen
Ok, I just commented out my .sbclrc and SBCL will start, but then I just tried starting it in emacs, and it is telling me "The value SWANK/BACKEND is not of type (OR FUNCTION SYMBOL)"
0:54:04
mrcom
Are there any freenode archives (in particular, #lisp) which allow search over an entire year's IRC?
0:58:00
mrcom
Xach: , or anyone, any thoughts on updates to "Go-to libraries"? (Zach's 2015-04-06 blog post)
1:10:30
rumbler31
oh. can you load sbcl without init, then manually load and start swank, then use slime-connect manually?
2:39:12
rumbler31
after you do that, start with an empty sbclrc, and follow the quicklisp setup steps
2:43:32
emaczen
rumbler31: When I install stuff from homebrew it just seems to work, and this problem got started since I tried to build SBCL from source myself
2:46:27
emaczen
on_ion: Yes, what I just followed similar advice using those immobile-code flags when you build SBCL from source with "sh make.sh"
2:52:29
emaczen
How do I uninstall everything that I built from source? This is no uninstall script.
2:52:53
rumbler31
if you didn't place the installation andywere, just blow away the folders you're compiling in
2:58:25
rumbler31
but its the only other thing I can think of about "persistent files" that might need to be cleaned up
3:00:16
on_ion
emaczen: hmm, is it full build? i just built from git yesterday with --fancy and worked great (debian)
3:11:52
emaczen
What versionare you using? I think it is the latest stable build? It was what was linked on sbcl.org
3:13:12
rumbler31
use that version you just made, to compile a source tree that includes your patch, or the removal of immoblie-code
3:13:44
emaczen
rumbler31: I mostly use SBCL and CCL, but I don't know how to get CCL fast enough for some programs I write that need speed
3:14:37
rumbler31
so are you also specifically passing in the lisp to use for compilation? I think you need to specify that to make.sh as well
3:15:47
rumbler31
if you don't specify I think it uses whatever sbcl is in the path, if that isn't he one you just made then you might run into trouble
3:17:11
rumbler31
but even then I don't know if make.sh knows to use the sbcl binary that you just built, it probably still looks in the path