(http://web.archive.org/web/20070315153349/lists.midnightryder.com/pipermail/sweng-gamedev-midnightryder.com/2005-August/003798.html)

I’m storing it here for no other reason than it interests me and I don’t know how long it will remain up for on the original site :S

19/01/2006

At last, some more details on what Naughty Dog gained from using Lisp to program their games:

Responses inline:

>From what I gathered, what GOAL had that I’d like:
>- fast iteration (due to the listener)

Well, fast iteration times weren’t merely due to the the listener – that was
a nice touch, but only the tip of the iceberg.  We could basically
dynamically link per-function or variable.  Effectively, you could hit a key
while working in the IDE, and whatever function the cursor was on would
instantly get compiled, sent across the network to the TOOL, linked and
dropped into the game while it was running.  The whole process took a
fraction of a second.  You could also do the same per-file.  This feature
was sort of like Edit and Continue, but you didn’t have to broken in the
debugger – it could be done while the game was running.  This was insanely
useful for programming gameplay, physics, and fx, as well as prototyping,
visual debugging (just drop in some debug spheres or prints while you have
the game in some interesting state), etc.  We also used it for dynamic code
streaming – so only a fraction of the executable code was loaded at any
given time (to conserve memory).

It’s astonishing at how much this ability changes your coding style – you
gravitate to a much more iterative development process, and write small bits
of code that evolve the codebase while the game is running.  It’s obviously
also much better for robustness and reliability, because you’re testing out
each little piece of code as it gets added (as opposed to writing a whole
mess of code and trying to get it compiled and debuggged).  As for bugs,
often you can fix them while the game is running and test out your changes
instantly – this makes bugfix verification much easier.

>
>- more performant generated code (I honestly find it hard to believe that
any small group of people (at least the ones I know!) could write a better
compiler (as in generates faster code) than Microsoft or GCC4 at this point,
but, perhaps compared to an ancient/early version of GCC on PS2?). Or is it
more that if you expressed certain things in
C++ that they’d be inherently slower?
>

On the performance front, the actual optimizing compiler really wasn’t that
great.  In fact, in most cases it generated code that was clearly inferior
to GCC (it did take advantage of native register sizes, but overall very
little development time was spent on the optimizer).  However, due to GOAL’s
seamless and tight integration with assembly, this wasn’t as much of a
problem as it may seem.  On the PS2, most heavy, performance-critical
operations you’re going to want to delegate to the VU anyway.  Writing
assembly code was awesome in GOAL – it unified the assembly syntax across
all of the disparate PS2 processors (with the same syntax as GOAL), you
could freely intermix assembly from all of the processors and normal GOAL
code, and it could access standard GOAL member variables, plus you had all
of the macro facilities, etc, to do stuff like automatic loop unwinding,
code pasting, etc.  It was pretty typical to write a bunch of GOAL code and
just replace the inner loop with some macro-mode VU code for nice
performance gains.  This was a pretty easy task given the fact that all
assembly code had the same syntax (no MSVC “__asm” blocks or GCC’s extremely
awkward intrinsics), and it took advantage of the register colorer).

As for the other stuff, like co-routines, etc – one of the salient points
here is that GOAL was extensible.  Since we wrote the language in-house, we
were able to extend it when the demands of our game required it.  This adds
a whole new dimension to problem solving – when you aren’t constrained by
the limitations of the language, you find that many problems become a lot
easier to tackle.

It was this ability to shape GOAL to our needs that allowed us to develop
features like coroutines, state inheritance, variable overlays, unique IDs,
etc.  This really was one of our key advantages in working with the
language.

That said, GOAL wasn’t exactly designed with language extensibility in mind.
The macro syntax was quite powerful, but fell short of really allowing you
to add stuff to the core language, like new tags and keywords.  This meant
that a lot of new features had to be implemented by extending the actual
compiler, which was unfortunate – it was simply too time consuming to do
later in the PS2′s lifecycle.

In the future, I think it is absolutely critical that any game development
language is built with extensibility in mind.  This would imply both a good
macro syntax (for extensible code generation), compile-time reflection,
syntax extensions, and also the ability to add new tags, qualifiers, and
keywords (facilities that tools like OpenC++ try to provide).  After all, a
DS developer has a very different set of problems to solve than a PS3
developer, and a lot of these problems can be greatly aided by the
programming language (managing the many processors on the PS3, for example).

[Here's some sample GOAL code from a later email message in the same thread]

Okay.  Here’s some old code I dug up from Jak3 that uses regular GOAL code,
EE assembly, and VU0 assembly in macro-mode.

A few words of warning:
The syntax takes a while to get used to (it is LISP, after all), and this
function is sort of weird (it’s a particle system callback function – the
assembly is there because I needed to convert some data from fixed point).
Also, it’s been over a year and a half since I wrote this, and it was a
quick one-off for a weapon effect, so it’s a bit messy.

Most of the code here is GOAL code, except for the code in the rlet.  (‘let’
is the way you declare variables in LISP, and rlet specifically guarantees
that variables will be bound to registers).

(defun sparticle-red-2-converge ((sp-system sparticle-system) (particle
sparticle-cpuinfo) (spvec sprite-vec-data-2d))
“Red gun 2 charge glow effect”
(let ((control (the sparticle-launch-control (-> particle key)))
)

;; figure out the offset from the player’s position
(let ((origin-vec (stack-vector))
(offset-vec (stack-vector))
)

;; retrieve offset vec off of userdata, omega.  We only had 8 bytes
for data storage,
;; but needed to store a full vec3, so we packed it into fixed
point.
(let ((combined-vec (new ‘stack ‘vector4w))
)
(set! (-> combined-vec x) (-> particle user-uint32))
(set! (-> combined-vec y) (-> particle data 0))

(rlet ((offset :reg vf1)
(preconverted :reg vf2)
(iconv0)
(src)
)
(l.q src combined-vec)   ;; GOAL->EE
(mer.vh lo iconv0 src r0) ;; EE
(sr.vw iconv0 iconv0 16)  ;; EE
(m preconverted iconv0)   ;; EE->VU0

;; (format *stderr* “Value is ~d~%” iconv0) ;; GOAL

;; VU ftoi15
(cvt.s.w offset preconverted :fixed 12) ;; VU0

(s.q offset offset-vec)  ;; VU0->GOAL
)
)

;; figure out distance
(let ((time-remaining (-> particle timer))
(distance 0.0)
(tt 0.0)
)
(set! tt (/ (the float (-> particle timer)) (the float (frame-time
90))))
(*! tt tt )
(set! tt (- 1.0 tt))

(vector-normalize! offset-vec (lerp PARTICLE_RED_2_CHARGE_SIZE
(meters 0) tt))

(vector-copy! origin-vec (-> *target* gun fire-point))

(set! (-> spvec trans x) (+ (-> offset-vec x) (-> origin-vec x)))
(set! (-> spvec trans y) (+ (-> offset-vec y) (-> origin-vec y)))
(set! (-> spvec trans z) (+ (-> offset-vec z) (-> origin-vec z)))
)
)
)
)

Scott Shumaker
Lead Programmer, Naughty Dog