The problem with having your code all in a software image is that the second you do this you lose a large existing toolset that people have invested a lot in mastering.
Just think of everything that you know how to do which involves a filesystem. I regularly use editors, grep -r, cd, edit in one directory and run unit tests in another, revision control systems, etc. People have invested a lot of energy into these tools, and have strong opinions about which ones they prefer and why. (Perhaps you prefer an integrated IDE. Doesn't matter, my point remains.)
As soon as you move to an image based system, your entire toolset no longer works within the inside of that image. It is, of course, a programmatic environment and replacements for every filesystem manipulation you are familiar with are available. Sometimes those replacements are better (though better is often just in the eye of the beholder). But they are guaranteed to be different. And as you move from one image based language (say, a Lisp dialect) to another (eg Smalltalk), they will be different again.
By contrast if you move from one filesystem based language to another, you will still have a learning curve, but there is a large toolset that you do not have to relearn.
Just to be clear, I'm not advocating giving up the filesystem :) I wholeheartedly agree that too much work has been done there to just forsake the toolchain, but that doesn't mean you can't creatively compromise to get the best of both worlds. Exactly how that will play out will be really interesting to see I think.
We've been working on some big changes to the architecture of Light Table that should set us up for figuring that out in the future. A blog post is probably coming next week about some of that.
"The problem with driving a car is that you lose almost all the skills you obtained by driving a horse carriage".
Yes, the image interop part is a bit more difficult to solve. I'm not saying it's exactly like the switch from carriage to car, but I'm saying that some changes are worth losing existing skills
Consider that image-based environments have been losing ground to file-system based ones for years.
Consider also that core ideas that make image-based environments good can be separated from the image-based system and moved into non-image based systems. Two examples that I would give for Smalltalk are Ruby's borrowing of Smalltalks OO model, and various IDEs borrowing the idea of a refactoring browser.
Borrowing the other way is much harder. For instance consider the distributed version control patterns that git enables. Image-based software development communities are sufficiently small and fragmented that they are unlikely to all duplicate the functionality of git. And because of the internal differences, they each would have to reinvent it. By contrast once git became available, it was immediately available for use with C++, Java, Python, JavaScript, Haskell...
There are advantages to image-based environments, but do they outweigh the disadvantages? Before concluding that they do, you might want to re-read Worse Is Better.
I think you hit the nail on the head. Build systems like GNU Make pull information about the file system to find out what has been updated. All we basically have to do is to
1) move from an information pull to a push. (Sidenote: This is something that 'kind of' works in OSX GUI, although it seems to me that 10.7 has broken this functionality for certain usecases. E.g. I'm connected to my host mac with my linux vm through SSH and when I've updated a file from within that VM, the host Finder windows get updated automatically and Preview.app windows get updated when I click into them).
2) Extend the functionality to runtimes as well, i.e. enable hotloading like in the new Unreal engine.
Surely it is possible to have an 'image-based system' where can also work with 'the existing filesystem toolset' as required. Its a bit of complexity in bridging the gap, but I don't think necessarily a show-stopper (you might disagree with me here - happy to discuss if you are interested). I agree its a big issue, but not necessarily that its a binary choice as you describe it.
> The problem with having your code all in a software image is that the second you do this you lose a large existing toolset that people have invested a lot in mastering.
Intersystems Caché.
There's a CACHE.DAT, which sits on the filesystem as a giant monolithic file. It's just a big block device containing a B-tree. In some nodes of that B-tree you'll find binary code. There are specialised tools to manage all this. In other nodes you will find data. There are specialised different tools to manage all this.
There is the one virtual machine and it only runs one language fairly well, called MUMPS. There is an object-oriented addition to it. It's basically a huge bunch of macros that compiles down to... MUMPS.
You can, in a hackish way, sometimes call specific functions from a DLL but that's as far as it gets. However, effectively...
No libraries. At all. And that's the biggest pain.
No libraries. I want you to let that sink in. All those things on Github and SourceForge and everywhere you, simply put, can't use.
No libraries. Except the few that the vendor, Intersystems, chooses to eventually let you have... with often broken functionality. There will never be any UI toolkit or Javascript interpreter or HTML engine or nice XML parser or any of those things. There is a crappy JSON parser, but that's about it in terms of recent niceties, and it's only there because some other customer would have paid for it.
No libraries. And that's the biggest problem with these software image things.
No libraries.
The biggest advance that we managed to get in was installing hooks in the proprietary source code editor (no vim or choice of text editor here) to export and import the code from disk. On save and load. It helped a lot to be able to get all the classes into actual files that we could use SVN with and made things so so much easier.
Software images might be a good idea in theory.
But in practice, you are throwing out your file browser, your text editor, your libraries and everything, everything, else.
Something I've always wondered is how image-oriented languages like smalltalk handle deployment, version management, etc. How do you make a frozen, repeatable bundle? I realize that may be the wrong question, that those things are irrelevant, but if so, why are they irrelevant? What is it about those environments that compensates?
Just because you have a feature, doesn't mean you have to use the feature. When I did it, I just started a fresh image and loaded all my new code (checked out from the repository) into it.
Or sometimes I probably just used the existing image. I don't remember. Unless your code depends on something being not defined, or you deleted a lot of old code, it wouldn't really make any difference.
If I ever had a lot of code, I could have built the image offline (though Lisp compilers are pretty fast). If I had a lot of code to build and database migrations to handle on a live database, it could have gotten hairy. I guess that's just the price I would have had to pay for the added flexibility of being able to build my image offline. IME, complex database migrations in production can get hairy on any language/platform.
It's not really that much different from Python or Ruby, is it? They have a "repl" and an "image", except they're much weaker and not formally defined, or portable, or able to be saved or loaded, or anything like that. But if Python 4 was released today and Guido said "there's now functions to load and save images!", it's not like every Python deployment in the world would stop working. Everyone would have a new mechanism they could use, or not, as they liked, but it wouldn't affect their existing procedures.
Originally they used change sets. That didn't scale particularly well. Monticello is a distributed versioning system that was later developed for Squeak smalltalk. http://wiresong.ca/monticello/ . The Cuis project, a fork from squeak, imports/exports projects from your filesystem, regardless of whatever external VCS you use. http://www.jvuletich.org/Cuis/Index.html
I guess I don't understand where the line between code and data lies, then. Is there a store for variable data that is separate from the image?
EDIT: To clarify, I know that Lisp and Smalltalk have a fuzzy-to-nonexistent line between code and data. What I want to know is how you'd manage that in practice. If I'm changing an existing piece of software, I don't want to expose production users to its incomplete state, so I can't change the running production site directly. (Right?) But if I'm developing locally, or even in a staging version, I don't want to transfer my test/sample data.
Maybe a better question is, "What is a typical workflow like for modifying production software?"
> Maybe a better question is, "What is a typical workflow like for modifying production software?"
In a modern Common Lisp web server set up, you'll likely have a SWANK connection available for debugging and patching. You will have to write live-patching code to correctly lock, drop/unbind old definitions, rebind new definitions, and unlock. It will likely remind you of a DB migration script.
For a non-live upgrade, you'll have a process that starts up the new server image, gets it ready to receive connections, then switch the routing & drop the old server.
It's not too different from other systems, IMO.
To quote an old lisp saying, code is data is code.
If I want to do an image deploy, what I'll do is load the libraries into the image as part of the build, set up a 'main' function that the executable will trigger, then main will read any config files and go. This image will be an actual executable for the OS. The receiver will execute the executable and not know the difference.
edit: In terms of version control, each deploy to production would generate a tag.
In languages like Smalltalk and Lisp, there often is no difference between code and data. The famous Lisp parenthesis exist exactly because your source code is both code and data (specifically, nested lists).
Though if you were making a program that should be deployed at the end user you would properly make it read and write files or communicate with a server over the internet and obviously that kind of data cannot be put into the image.
I think the idea is that the code and data are dealt with and versioned in the same way (a common saying about Lisp code is that the "code is data.") I haven't actually used any of the environments mentioned in the article, but I believe that the code, as well as the entire state of the running program, can be snapshotted and versioned in these environments.
That's great if you only have a single developer. Larger teams working together are going to have issue with this.
I think that may have been part of why the Lisp Machines died - not only were they obscenely expensive, but they were extremely expensive single user machines. This is fine if your entire development team is one MIT supergenius but becomes problematic when you merely have a handful of rockstars.
It's always refreshing to see someone explore the work of those who have had similar concerns or solved similar problems. Whether they have independently chosen a similar direction or are standing on the shoulders.
Provides a good reference for new comers wanting to understand what approaches have been explored before.
We live in exciting times. The confusion of (programming) tongues is in full swing; the CPU clock wall is with us; high level features such as closures and GC now have fairly efficient implementations - it seems there are now enough transistors to deliver much of the abstraction that was envisioned by the founders but gave way to pragmatic concerns in the early microcomputer era.
I am looking forward to IDEs that manage and simplify complexity, and hardware ISAs that throw a few more transistors at things like STM, tagged data and memory management.
I can see a future where I can edit code in a language foreign to me, but see an instant side-by-side (editable) version in a dialect I'm more comfortable with. Switch in another view and drag and drop states in my state machine via direct manipulation, and so forth.
This really got me thinking about the ideas that we can mine from old, neglected systems that lost the marketing battles and are simply considered obsolete now.
So one of the things I've been working on in my 'spare' time is a simple training computer. One of the things I have been very careful to do is document what sources of information and inspiration I'm using in that design/implementation (all pre-1992) in order to insure that should someone try to throw a patent troll lawsuit at it I would be well equipped to defend. Sad really.
talk about blast from the past, I spent a few years looking at a Symbolics machine, though I rarely saw that Genera login, the machine stayed up for days on end.
light-table mode for emacs/common lisp would be neat. As a schemer I think what would be really awesome is to see light-table merged with the emacs/guile/geiser work. Geiser is also trying to resurrect the lisp machine.
I had a Xerox 1108 Lisp Machine from 1982 to about 1987 (and for one month, I ran Smalltalk on it rather than Lisp).
Yes, it was awesome compared to using a timeshared VAX (or whatever), but as fantastic as my 1108 was for its time and no matter how much I loved it at the time, I like my development environments today a lot more. Examples: development and debugging of both server side and client side code (GWT on client, something REST full on the server side) with IntelliJ Java debuggers open on both client and server code - hopefully you may only need this once a month but useful when you need it. Repl based development with a smart IDE (like Clojure in IntelliJ or Ruby in RubyMine). Interactive Scala worksheets in the Eclipse/ScalaIDE (sort of like Chris's Light Table).
In addition to a lot of NLP work on my 1108, I was also just getting into neural networks back then and my NN simulations ran like mollasses on the 1108. Any of my laptops are many orders of magnitude faster for just about anything I do than my 1108.
chris - In the 8/17 post you said there would be a big announcement coming up soon. Any updates on that? I think there was some speculation that it would be Python support - that would be awesome!
[please refute, not just downvote] Just because something can be done doesn't mean it should be done (maybe it should be done, but for some other reason).
I'm suggesting that barriers to modifying systems have had an accidental benefit of favouring standardization.
The basic problem is that it's hard to create systems (APIs, generalization, documentation, test-cases, integration with everything else). Brooks claims it's 9 times as much effort as making something that "works for me". If you can't amortise that over many users, it's usually not worth doing the job properly.
But if it's just for you, you don't need all that polish, right? Well, no. If you are standing on the shoulders of giants, it's really helpful if those giants are stable, function as you expect, interact correctly with other tools, and have clear documentation for when you've forgotten the details (writing it yourself doesn't make your memory perfect). SOTSOG is really important for building complex systems, because it enables you to push down details to a lower level of abstraction, so you have less clutter to deal with. It's getting the abstractions right that is so difficult (we've all seen it done wrong).
There are huge advantages to using standardized systems, that have been improved, tested, documented etc by others. If you want to do something different, it's very helpful if you can start with standardized components around the new part, so you can just concentrate on the interesting stuff, and not need to build the whole world from scratch. Of course, the existing stuff is not perfect - the question is whether it's worth it for you to spend the time needed to make something better.
Some issues in assessing this argument: These are ideas have been around for a long time, but haven't taken off. Highly modifiable systems like lisp, despite having clever and devoted users, lack the standardized libraries of less flexible languages like Java.
But there is one kind of area where such systems are absolutely brilliant - exploration at the systems level. Many (most?) programming language ideas have been developed in lisp, even though did not become popular in that form, because lisp is great for language exploration, but not for standardization. Similarly, an OS that can be modified on the fly, is great for exploring OS ideas. You can iterate quickly, see the results and so on.
Of course, it seems possible that one could have it both ways - a highly flexible system that also had standardized components. But that doesn't seem to happen. Flexibility and standardization are natural enemies, both in differences in the tools that support them... and the kinds of people who use those tools.
BTW: There's a middle way. Create tools that can be standardized on. i.e. libraries, frameworks, engines, DSLs. Light table could be perfect for that... but people who use it may want to make tools that are just as flexible as light table, and so don't support standardization.
Just think of everything that you know how to do which involves a filesystem. I regularly use editors, grep -r, cd, edit in one directory and run unit tests in another, revision control systems, etc. People have invested a lot of energy into these tools, and have strong opinions about which ones they prefer and why. (Perhaps you prefer an integrated IDE. Doesn't matter, my point remains.)
As soon as you move to an image based system, your entire toolset no longer works within the inside of that image. It is, of course, a programmatic environment and replacements for every filesystem manipulation you are familiar with are available. Sometimes those replacements are better (though better is often just in the eye of the beholder). But they are guaranteed to be different. And as you move from one image based language (say, a Lisp dialect) to another (eg Smalltalk), they will be different again.
By contrast if you move from one filesystem based language to another, you will still have a learning curve, but there is a large toolset that you do not have to relearn.