freenode/#lisp - IRC Chatlog
Search
22:09:35
stacksmith
I learned a tremendous amount when I started out - by reading Naggum transcripts. It was never boring, although somewhat painful on occasions.
22:11:36
dim
https://tapoueh.org/blog/2018/03/object-relational-database-management-system/ if you're interested, comparing PostgreSQL type system / function overloading with CLOS generic functions
1:18:32
Bike
you put the defun in your repl, and then the next line try to call it, and it doesn't work?
1:24:52
Bike
gethashes is a function that gets a value out of some hash tables. it has no provision for writing into hash tables
1:25:07
Bike
there is no way for the compiler to look at your loop and figure out an inverse procedure, see?
1:26:28
Bike
it analyzes the form (gethash key table) and macroexpands into something that writes into the table.
1:29:50
Bike
you can also try (macroexpand-1 '(setf (gethashes foo "two") 2)) to see why that didn't work out.
1:34:40
Bike
if you're still thinking about pointers, keep in mind c has lvalues as a distinct concept. you can't get the address of a register variable but you can assign to it no problem.
1:36:10
jasom
johnnymacs: you know how you can doo foo[bar] = baz in C, in this case SETF is like the = and gethash is like foo[bar]
1:37:21
Bike
you can do (incf (gethash foo bar)), for instance, to increment a value in a hash table
1:38:08
stacksmith
johnnymacs: unlike forth, where cells are pointers and @ and ! read and write to them, in Lisp mentioning a variable fetches it by default. setf sets it. setf can set a lot of complicated forms like gethash. You can also write a function to let setf know what to do with your gethashes.
1:53:08
Bike
so, lisp has assignment, but it is not pointers, and it is not "hacked to behave like pointers" or whatever
1:54:22
Bike
they aren't! all you're trying to do is set values in a hash table! that is easy to do with setf gethash, you just want to deal with pointers instead for some reason
1:55:19
Bike
yeah, and you already wrote that. now you just need to write a (setf gethashes) that does a loop the other way or whatever.
1:56:48
johnnymacs
If I am going to modify setf to handle gethashes the only way I can do that is with an array/hash table of functions or nested conditionals
1:58:04
Bike
this gethashes thing, it's defined so that (gethashes table a ...) = (gethashes (gethash a table) ...), right?
1:58:41
johnnymacs
Well what happens to the speed of setf as the number of cases setf can handle increases
2:01:12
Bike
(setf (gethashes table ...) new-value) will expand into (funcall #'(setf gethashes) new-value table ...)
2:02:02
Bike
of course, for most uses i imagine the number of keys will be fixed, so you could write something to do it without a loop
2:21:01
stacksmith
johnymacs: it's great that you jumped in and are coding, but you have a bit of a learning curve. Rather than fighting it, you need to get to the point where CLHS makes sense. Also not bitch that CL is getting in your way. It is your brain that's in the way of accomplishing tasks, not CL.
2:25:01
stacksmith
Bike's code should pretty much to the basics. Also read about Lisp macros - I think you may not fully appreciate them. They are kind of like Forth immediate words - they run at compile time and transform the code to your wishes.
2:42:04
jeosol
anyone encounted (sb-vm::tal-call-symbol ....) error while running multiple threads. My code runs okay when I run serial, but I switch to parallel, I get that (sb-vm::tal-call-symbol ...) error and then says a function (defined by macro) is undefined
2:49:47
jeosol
This is the link to the section of the code. Just create pastebin account. Pardon the upcase, as the code is generated from macro
2:50:41
jeosol
First of all, I will say I am not an expert lisp programmer, but I am writing an application in Lisp using SBCl for an optimization challenge with deadline in a few weeks
2:51:43
jeosol
things run serial without problems, but when I switch to parallel to gain some speed up, I get problems with GET-CPG-CELL-FACE-X- not being defined.
2:52:27
jeosol
I just simply abort. I have never been able to run the code in parallel lots of macro generated functions.
2:54:37
jeosol
the problem is an optimization problem. I am doing a bunch of F(X), where X is a vector of solutions. There can be many solutions in one iterations (think stochastic optimization algorithms), I can evaluate serially, but can get good speedup
2:55:14
jeosol
if I do things in parallel. Serially evaluation runs okay, but when I switch to parallel, I get some that SB-VM issue. I am using lparallel library
2:56:10
jeosol
@pierpa what is background/domain? the data for the challenge is not public in the sense that a request has to be made to get the data
2:56:54
jeosol
and solving the problem needs access to some other black box modeling tools. I can provide more details offline if needed
3:00:42
jeosol
@Bike, this new paste shows the restarts I get when there is an error https://pastebin.com/bJqGkceM
3:02:15
jeosol
the calling function is defined in the same package as the function that it is complaining about, except that the latter is defined by a macro
3:03:45
jeosol
@Bike, it is weird, runs okay, when serial, but with parallel it blows up. there is something in the code base, that is not correct. I thought it was race condition related so I ran parallel option but with one worker per batch, same error
3:08:49
jeosol
@pierpa, the problem is really challenging, each function takes 14 mins on my machine. and I need to run hundreds or thousands of function evaluations, so parallel is definitely the way to go
3:08:54
stacksmith
jeosol: are there any global variables in play? They may not be the same in a different thread...
3:10:00
jeosol
Thanks @stacksmith, appreciate the help. That defvar is in the same package. It is a vector of other functions defined by a macro.
3:10:37
stacksmith
I don't know if that's sufficient to be in the same package... I would print it and make sure...
3:10:52
jeosol
I posted an earlier problem, I got recommendations (I think it was @Bike, not sure) to use that instead of constructing function names dynamically using intern and format ..
3:11:14
jeosol
I posted an earlier problem, I got recommendations (I think it was @Bike, not sure) to use that instead of constructing function names dynamically using intern and format ..
3:11:35
stacksmith
It's fine to use an array, it's just that it's global. So when a new thread starts, it is probably nil.
3:13:40
stacksmith
Inside that thread that is. You may need to rebind it in whatever function that starts a new thread in lparallel.
3:20:32
jeosol
in the debugger I get this error: COMMON-LISP-USER::GET-NORMAL-CPG-CELL-FACE-X-1 is undefined
3:20:51
jeosol
Not sure why it is looking for that function in the CL-USER package when I run in parallel?
3:28:42
stacksmith
jeosol: If you create a new thread and output to *standard-output* you will crash, because inside the thread *standard-output* is nil (or maybe unbound, I can't remember).
3:30:09
Bike
not directly, but if it's looking for a symbol in the wrong package that seems like a good place to start looking for problems
3:30:39
stacksmith
Honestly, the first thing to do is print the array and see if it has functions in it...
3:45:45
pierpa
jeosol: wait, "defined by a macro" means that your macro is creating the symbol you use as the function name?
3:45:56
jeosol
@stacksmith, my application runs and writes statement to the *standard-output*, e.g., results from computations, etc.
3:46:44
stacksmith
Are you sure that the output comes from inside the thread - or from main thread after the computations are done?
3:47:50
stacksmith
Also, if you put the whole program into a thread, it's possible that some intialization function causes macros to expand inside the thread, with CL-USER as the default package...
3:48:48
jeosol
@stacksmith, no, I am not saying you are wrong or anyone. I need suggestions on what the problem could be
3:49:35
stacksmith
jeosol: if you want to rule out the issues I proposed, just output the values of *package* and your array from inside the thread...
3:52:55
stacksmith
Bordeaux-threads docs state "Global bindings are shared between threads: the initial value of a global variable in the new thread will be the same as in the parent, and an assignment to such a variable in any thread will be visible to all threads in which the global binding is visible." - but I always have to rebind *standard-output* or it does not work.
3:53:59
stacksmith
So if any of the globals at thread startup happen to be bound, the values are lost.
3:54:51
stacksmith
Hence, I suspect that package may be bound, and inside the thread it's what it was originall - CL-USER.
3:55:33
pillton
I'm not sure how bordeaux-threads can make such a claim given that it is up to the implementation to decide what happens with the environment when a new thread is created.
3:56:59
stacksmith
I would imagine that threads are started way deep, in a place where bindings are not in scope.
4:00:07
Bike
you could have all special bindings be in thread local storage. then there'd be no global bindings.
4:02:55
stacksmith
Then you would have to keep track of all special bindings separately. Do implementations do that?
4:04:30
stacksmith
Well, whatever way symbols are associated with values - you would need to keep a separate list and upon thread creation recreate it in the new TLS buffer - and you would still wind up with the initial global values...
4:06:56
stacksmith
Except, the outer bindings would contain the entire rest of the system, requiring a full recompile every time anything is changed - at least the way SBCL currently works...
4:09:23
rme
Typically, implementations make certain special variables thread-local (such as *print-base*, etc.). Other special variables see a global value. In CCL, there's a form called defstatic which is like defparameter, but asserts that the symbol in question will never be given a thread-local binding.
4:12:49
Zhivago
Personally, my preference is to use (possibly very light-weight) processes rather than threads, so that this problem doesn't exist.
4:14:38
jeosol
This is a pain, not resolved, yet, but may be I have to rethink how I structure things
4:16:07
jeosol
I will keep hacking at this as I need the parallel functionality. Thanks for all your comments. I am monitoring the chat for hints ... :-)
4:16:36
stacksmith
jeosol: have you tried to output *package* from the environment in which your macro runs? Is it possible that you have an (eval-when ...) initialzation that somehow winds up inside the thread?
4:20:21
stacksmith
I think so, if the thread initialization is around the macros. They run in a new context in CL-USER package
4:24:49
stacksmith
I try to spawn new threads after the system is loaded and all compilation and expansion is done...
4:34:49
stacksmith
I think so, but you may want to decide if you want to truncate or deal with floats or fractions, etc.
4:41:55
jeosol
@stacksmith, I don't start the threads in eval-when. I use that to define the functions when I load the system. I think the problem is related to *package*
4:48:51
jeosol
My application is very large so I am using asdf:package-inferred-system for all my files and I have to manage things carefully. See the updated repo as of January 2018 https://www.youtube.com/watch?v=sgdiRh-alfQ
4:48:56
pierpa
that EXPORT will not make the symbol accessible in any package apart than it's home package
4:49:21
stacksmith
Do you need to have the macros in eval-when? It may not be the issue, but is there a reason?
4:50:57
jeosol
@stacksmith, I only use the threads for the function evaluations. Multiple calls SIMD type, F(X), where my X's are different CLOS objects
4:52:00
pierpa
jeosol: your system is the one shown by the video, or your system is the thing which produced the video?
4:52:06
stacksmith
Well, I still think that the first thing you should've done a couple of hours ago when we started this - print or stick the value of *package* somewhere and look at it.
4:54:10
jeosol
There is a visualization for SBCL code base with GOURCE on youtube but not sure how updated it is. There are others for LINUX, python, etc
4:57:49
jeosol
The tool shows evolution and complexity of GIT repo. I was just fooling around with options to GOURCE and added elasticity to the branches etc
5:01:24
jeosol
@stacksmith. The code base does not focus on a single problem but a series of interrelated problems in a domain. The part on optimization is focused at that challenge I mentioned. There are parts doing some small ML type workflows, etc
5:02:05
jeosol
but my focus for the last few moments is in the optimization part to get it to work for that challenge problem.
5:03:53
jeosol
I asked a question yesterday here about web frameworks in CL to use. I wanted to build a web application to have public access. I know a bit of hunchentoot and cl-who, html, javascript, etc, I will do this later, but for now, I just run within slime
5:05:54
jeosol
@stacksmith, like I said before, I am not an expert on a lot of things. The part of Cl I understood the most is CLOS part because my code is essentially OOP which is easier to manage for me. How things work internally, is not all clear to me so I learn as I go
5:06:04
stacksmith
jeosol: I've been up for two days and getting fuzzy, but if your export is in the macro definition, you probably want (export 'name ,*package*), no? Splice in the package as it exists at expansion time, not when the macro was defined...
5:07:55
stacksmith
Wait, that's backwards. You want no comma, the package at expansion time, not at definition time. Can someone look at this because I am getting cross-eyed...
5:08:02
jeosol
@stacksmith, thanks for the suggestions, you can get some rest man. I will go through the messages to review the messages again
5:09:40
stacksmith
Wait, it is right there. Without a comma it expands to *package* and gets its value then and there, with a comma it spliced in the package at macro definition time.
5:10:17
pierpa
you should either create the symbol in the right package with (INTERN "NAME" the-right-package), or IMPORT the symbol in the right package later. Your EXPORT does nothing.
5:11:31
jeosol
Since I use asdf:package-inferred-system and essentially one-file defsystems, I usually import the functions one.
5:12:10
jeosol
For the error I am getting the caller and called function are in the same package. Everything blows up when I go parallel function evaluation, otherwise, it runs without problems.
5:12:57
stacksmith
jeosol, just to be clear, macroexpanding your macro in REPL and having it expand in its eval-when environment during startup is likely very different. That's why I've been urging you to figure out a clean way to see what *package* is, at that point and go from there...
5:13:32
pierpa
I suggest reading carefully the packages chapter of CLTLII, or alternatively the relevant parts of CL recipes.
5:13:33
jeosol
@stacksmith, ok let me do that. You men when I am running the application, to print the value of *package*
5:14:39
stacksmith
yeah, but be careful about when you do it. You want to modify your macro to push the package onto a global, I think - because it might expand more than once....
5:14:49
jeosol
Thanks @pierpa, I need to get a copy of that book for sure. When my system got large, after a chat with Fare and switching to one-file system, it is painful but manageable
5:41:24
jeosol
ok, I asked because it you are working with time and dates, it is easier to use one of the time-date librarieis
5:41:48
jeosol
Which one you use depends on what you are doing, time resolutions, if you want to track local-time, etc
6:32:44
stacksmith
didi: not as good as "In theory, there is no difference between theory and practice. But, in practice, there is. "
6:35:56
stacksmith
Or perhaps my favorite by Alan Perlis: "Syntactic sugar causes cancer of the semicolon."
9:11:20
flip214
because long lines that get broken might be interpreted as first line + syntax error or, worse, two individual lines
9:37:37
dim
what would you recommand to a retiree using windows who wants to hack some common lisp, in terms of setup? I'm thinking ACL or LispWorks might make sense in that case, right?
9:46:45
dim
yeah I told the guy to use Emacs if he already knows how to, or if he's ready to take on the learning curve
9:47:14
dim
the other options are for if he wants a windows native IDE for CL, I guess, where you can click and discover things
9:47:55
dim
Shinmera: with some luck this guy will to, but I just don't know him and won't be able/willing to offer any guidance other than an email sometimes, so...
9:49:42
jackdaniel
being able and willing two are two different things. if I had a choice I'd dump emacs today despite knowing it reasonably well
9:52:24
dim
oh? for me Emacs embeds my whole workflow, tailored the way I like it… I think the only reason I can work from a mac laptop is because I have Emacs and the underlying Unix tools easily available
9:53:55
jackdaniel
I agree it is a powerful tool etc etc, and when I'm installing Windows VM to test ECL usually I'm hacking from inside Emacs too
9:56:51
runejuhl
dim: maybe spacemacs? I've only tried it briefly, but it seems like a good introduction to emacs with a pretty sizable community
9:59:41
dim
I sent the email now, saying Emacs/SLIME first, then maybe ACL or LispWorks for having a windows like experience
10:00:34
dim
jackdaniel: oh and I think for hacking lisp I would like having a McCLIM based environement with some kind of graphics support and things, really, but well, it's not there yet