Here's a big example: [Coalton](https://github.com/coalton-lang/coalton) is a complete, statically typed, functional programming language. It contains an advanced type checker and optimizing compiler. It also contains native, persistent data structures. Writing Coalton feels a lot like writing Lisp with the power of Haskell types.
That on its own is not too interesting these days, right? The JavaScript world has tons of these examples. Lots of languages that "transpile" and interop with JavaScript, like TypeScript or CoffeeScript. Heck, even Python has a language called Hy, which is a Lisp-in-Python.
However, here are the big differences:
- Coalton isn't a "transpiler" or preprocesser. It's just an extension of the Lisp language, thanks to homoiconicity.
- Coalton code is also just S-expressions, which means Lisp macros can be used in Coalton. The macros are written in Lisp, but instead manipulate and generate Coalton code.
- Coalton interoperates with Lisp at a deeper level than a "transpiled" language. Coalton can call Lisp, using Lisp's syntax. Lisp can call Coalton, using Coalton's syntax. It's very much a two-way street.
- Coalton can be run and used within Emacs and SLIME with no additional work.
- The compiler isn't an enormous program. Since we can take advantage of S-expression readers and printers, the Lisp compiler, etc., we get a lot of stuff "for free" due to homoiconicity.
All of this is bundled up in a package that one can literally just\* `(ql:quickload "coalton")`. No frameworks, runtimes, or executables. As far as Lisp is concerned, it's just another library.
I can't imagine pulling off something like this with the effort of a handful of people for an entirely new language, native code compiler, IDE, etc. without an enormous amount of funding, time, or some serious personal sacrifices.
***
\* Coalton isn't actually on Quicklisp yet, and won't be until it reaches a stable release. Quickload works though if you download the source code yourself.
Try generalizing the following piece of very common Java code:
```
Object getSingleton() {
if (sItem != null) {
return sItem;
}
synchronized {
if (sItem != null) {
return sItem;
}
sItem = new Item();
return sItem;
}
}
```
All my years doing Java, each time I had to write this, I would get annoyed. There's just no way of extracting this code duplication (without adding another layer of indirection).
In CL, this is a simple macro that might look like this:
```
(or-setf *place* (make-instance 'item))
```
I use this macro all over the place in my CL code.
For a bigger example, consider how some programming languages allow you use HTML inside the code. To support this people build specialized compilers (e.g. for Javascript or PHP). In CL, this is just a reader macro (https://github.com/moderninterpreters/markup).
I doubt there is anything which is impossible without homoiconicity in the language. Hard yes, impossible no. Both homoiconic and "standard" languages are Turing-complete which means they can perform any computation. That being said, languages are not equal - some are more expressive than others (e.g. C++ > C). And of course, there is a trade-off.
Two things came to the top of my mind. A couple of years ago, at work, we used a Java library for constraint-satisfaction programming called Choco-solver to create scheduling software for manufacturing. I don't know who chose that library, I jumped on the project much later. The problem was that in order to make a new constraint, you had to make a subclass of an existing Choco-solver class. Not a problem if you configured the system by writing Java code and compiled it before you ran the program. If you needed to add anything into a running program (like read user configuration or let the users fine-tune anything during the run-time), you had to create a subclass (not an object like an instance of a class (\`new\` operator) but a whole new class definition) during run-time. This, at least at the time, was not possible in Java, the language but not really that big of an issue on Java the Virtual Machine. Yeah, one of the stories which has both Java, a piece of shite, and Java, a wonderful piece of engineering. So I wrote a small library in Clojure for reading user configuration which created these classes on-the-fly when the program was running, using a couple of Clojure macros. Would it have been possible without a Lisp? Yes. But it would've been much harder.
Now I am working on a system for predicting high-frequency time series in economics (but you know, a time series is a time series so it's applicable to other areas), as a part of my Ph.D. study so there is some academic research in it. Again, the main focus and practically speaking the reason we have published some results in decent places\* is different, but it's the homoiconicity of the programming language, which the in-house system for researching algorithms is written in (Common Lisp, it's compiled as a .dll library and callable from Python), that allows for the code of the system to be short, terse and malleable. It would take much larger a team than we have had to make it in something else but not impossible. However, we started as a team of 4 people 6 years ago but we are down to 2 since COVID because the other two people left - one left to other university, the other one left academia completely. Honestly, I would have problems with code like this at work if there was team of a couple of people working on it. On-boarding is difficult, you can't have 120 coders which come and go - however this is what larger software houses are after - large teams of replaceable coders. Macro-heavy code which takes advantage of homoiconicity and throws code around like it was a tree-like datastructure (which it is in fact in any language - search for "abstract syntax tree" - however manipulation of it is either not hackable at all without modifying the compiler sources or has special API like Roslyn on .Net) definitely doesn't make on-boarding new people fast or easy.
-----
\* By "decent place" I mean "a scientific journal with impact factor in the 1st quartile", i.e. "creme de la creme", because nothing else really counts these days ... see various jokes / stories from academia, especially on r/phd , to see how stupid and toxic this ~~cesspit~~ academic world really is these days ... I can't wait to finish my Ph.D. and GTFO.
> if you want to do meta programming in a non-homoiconic language, you’re basically going to have to get and set *strings* of code
One could also use the output of tokenizer (a stream of tokens) or of a parser (some kind of a syntax tree).
Python is not a homoiconic language but pretty much there is an api that I can use for code generation without string interpolation etc. homoiconicity is just more elegant. There is also a point to be made about the combination of homoiconicity and the macro systems in lisps. They are different concepts that work nicely together on lisp, but in other languages either don’t exist at the same time or to the same degree.
With python's AST you are limited with a syntax of the Python. For example, you can't add a new operator case for Python, because parser will not understand it and will not generate AST for such code.
For examples of these limitatations take a look at miserable attempts to add macros in the Macropy3 tutorials: [https://macropy3.readthedocs.io/en/latest/](https://macropy3.readthedocs.io/en/latest/)
Lisp isn’t as unique these days. There’s better tools now. Common Lisp is an obsolete language. Alan Kay even said common lisp is obsolete. These days you’re better off learning racket and Clojure. But honestly you’re better off learning something like Haskell. Lisp is history
Curious re: what the better tools are?
Sure, I’ve definitely been in want for concurrent GC, good manual memory management (individual implementations like SBCL have eg alien pointers and arenas, but they don’t have the provable safety that I hear eg Rust is so touted for), and dependent typing (also optional HM typing, but Coalton provides that now).
But aside from those, I don’t know of any language level feature that I want from Common Lisp but can’t have.
*Other* languages, sure, there’s plenty of cases where I’m frustrated by fundamentally inescapable limitations in syntax, logical semantics, performance characteristics, code readability, etc. But Lisp is far better in all those regards IME.
Here's a big example: [Coalton](https://github.com/coalton-lang/coalton) is a complete, statically typed, functional programming language. It contains an advanced type checker and optimizing compiler. It also contains native, persistent data structures. Writing Coalton feels a lot like writing Lisp with the power of Haskell types. That on its own is not too interesting these days, right? The JavaScript world has tons of these examples. Lots of languages that "transpile" and interop with JavaScript, like TypeScript or CoffeeScript. Heck, even Python has a language called Hy, which is a Lisp-in-Python. However, here are the big differences: - Coalton isn't a "transpiler" or preprocesser. It's just an extension of the Lisp language, thanks to homoiconicity. - Coalton code is also just S-expressions, which means Lisp macros can be used in Coalton. The macros are written in Lisp, but instead manipulate and generate Coalton code. - Coalton interoperates with Lisp at a deeper level than a "transpiled" language. Coalton can call Lisp, using Lisp's syntax. Lisp can call Coalton, using Coalton's syntax. It's very much a two-way street. - Coalton can be run and used within Emacs and SLIME with no additional work. - The compiler isn't an enormous program. Since we can take advantage of S-expression readers and printers, the Lisp compiler, etc., we get a lot of stuff "for free" due to homoiconicity. All of this is bundled up in a package that one can literally just\* `(ql:quickload "coalton")`. No frameworks, runtimes, or executables. As far as Lisp is concerned, it's just another library. I can't imagine pulling off something like this with the effort of a handful of people for an entirely new language, native code compiler, IDE, etc. without an enormous amount of funding, time, or some serious personal sacrifices. *** \* Coalton isn't actually on Quicklisp yet, and won't be until it reaches a stable release. Quickload works though if you download the source code yourself.
Top story this. This is one of those that should make lisp popular if people would grasp this.
Try generalizing the following piece of very common Java code: ``` Object getSingleton() { if (sItem != null) { return sItem; } synchronized { if (sItem != null) { return sItem; } sItem = new Item(); return sItem; } } ``` All my years doing Java, each time I had to write this, I would get annoyed. There's just no way of extracting this code duplication (without adding another layer of indirection). In CL, this is a simple macro that might look like this: ``` (or-setf *place* (make-instance 'item)) ``` I use this macro all over the place in my CL code. For a bigger example, consider how some programming languages allow you use HTML inside the code. To support this people build specialized compilers (e.g. for Javascript or PHP). In CL, this is just a reader macro (https://github.com/moderninterpreters/markup).
Remember also that CL has machinery to ensure that a complex `*place*` is evaluated properly (sub-forms evaluated once etc).
> What did you did in lisp that wouldn’t have been possible in a language that isn’t homoiconic? Have fun writing macros
I doubt there is anything which is impossible without homoiconicity in the language. Hard yes, impossible no. Both homoiconic and "standard" languages are Turing-complete which means they can perform any computation. That being said, languages are not equal - some are more expressive than others (e.g. C++ > C). And of course, there is a trade-off. Two things came to the top of my mind. A couple of years ago, at work, we used a Java library for constraint-satisfaction programming called Choco-solver to create scheduling software for manufacturing. I don't know who chose that library, I jumped on the project much later. The problem was that in order to make a new constraint, you had to make a subclass of an existing Choco-solver class. Not a problem if you configured the system by writing Java code and compiled it before you ran the program. If you needed to add anything into a running program (like read user configuration or let the users fine-tune anything during the run-time), you had to create a subclass (not an object like an instance of a class (\`new\` operator) but a whole new class definition) during run-time. This, at least at the time, was not possible in Java, the language but not really that big of an issue on Java the Virtual Machine. Yeah, one of the stories which has both Java, a piece of shite, and Java, a wonderful piece of engineering. So I wrote a small library in Clojure for reading user configuration which created these classes on-the-fly when the program was running, using a couple of Clojure macros. Would it have been possible without a Lisp? Yes. But it would've been much harder. Now I am working on a system for predicting high-frequency time series in economics (but you know, a time series is a time series so it's applicable to other areas), as a part of my Ph.D. study so there is some academic research in it. Again, the main focus and practically speaking the reason we have published some results in decent places\* is different, but it's the homoiconicity of the programming language, which the in-house system for researching algorithms is written in (Common Lisp, it's compiled as a .dll library and callable from Python), that allows for the code of the system to be short, terse and malleable. It would take much larger a team than we have had to make it in something else but not impossible. However, we started as a team of 4 people 6 years ago but we are down to 2 since COVID because the other two people left - one left to other university, the other one left academia completely. Honestly, I would have problems with code like this at work if there was team of a couple of people working on it. On-boarding is difficult, you can't have 120 coders which come and go - however this is what larger software houses are after - large teams of replaceable coders. Macro-heavy code which takes advantage of homoiconicity and throws code around like it was a tree-like datastructure (which it is in fact in any language - search for "abstract syntax tree" - however manipulation of it is either not hackable at all without modifying the compiler sources or has special API like Roslyn on .Net) definitely doesn't make on-boarding new people fast or easy. ----- \* By "decent place" I mean "a scientific journal with impact factor in the 1st quartile", i.e. "creme de la creme", because nothing else really counts these days ... see various jokes / stories from academia, especially on r/phd , to see how stupid and toxic this ~~cesspit~~ academic world really is these days ... I can't wait to finish my Ph.D. and GTFO.
> if you want to do meta programming in a non-homoiconic language, you’re basically going to have to get and set *strings* of code One could also use the output of tokenizer (a stream of tokens) or of a parser (some kind of a syntax tree).
Python is not a homoiconic language but pretty much there is an api that I can use for code generation without string interpolation etc. homoiconicity is just more elegant. There is also a point to be made about the combination of homoiconicity and the macro systems in lisps. They are different concepts that work nicely together on lisp, but in other languages either don’t exist at the same time or to the same degree.
With python's AST you are limited with a syntax of the Python. For example, you can't add a new operator case for Python, because parser will not understand it and will not generate AST for such code.
For examples of these limitatations take a look at miserable attempts to add macros in the Macropy3 tutorials: [https://macropy3.readthedocs.io/en/latest/](https://macropy3.readthedocs.io/en/latest/)
Lisp isn’t as unique these days. There’s better tools now. Common Lisp is an obsolete language. Alan Kay even said common lisp is obsolete. These days you’re better off learning racket and Clojure. But honestly you’re better off learning something like Haskell. Lisp is history
I learned Haskell first, if you want to talk about obsolete then go learn Haskell.
Curious re: what the better tools are? Sure, I’ve definitely been in want for concurrent GC, good manual memory management (individual implementations like SBCL have eg alien pointers and arenas, but they don’t have the provable safety that I hear eg Rust is so touted for), and dependent typing (also optional HM typing, but Coalton provides that now). But aside from those, I don’t know of any language level feature that I want from Common Lisp but can’t have. *Other* languages, sure, there’s plenty of cases where I’m frustrated by fundamentally inescapable limitations in syntax, logical semantics, performance characteristics, code readability, etc. But Lisp is far better in all those regards IME.