Does math love emergence?

There’s something I’ve been wondering about around functional and agile. It’s about creating solutions, creating knowledge.

I see agile geeks pulling toward functional programming, from young coders here in Ohio to famous folks like Brian @marick and @unclebobmartin. And I’m wondering what it will look like when functional marries agile.

I talk to functional/denotative folks who are excited by math, and clarity of understanding, and composability. And I’m excited by those things too. They’re very cool.

But it seems like functional’s missing out on something important that TDD offers. Specifically, emergent design. Emergence is how complexity works, isn’t it? I think what the functional / denotative people are saying is that we should use our awesome rational brains to understand a problem and work out a solution. Is that what they’re saying?

(This TED talk is about how we solve problems without understanding them.)

How do brilliant functional programmers deal with complexity? Do they have to figure it all out, without letting emergence help?

I’ve been thinking it’d be very cool to create a cafe or an un-conference to address this. How do we get all the juice out of denotative mathy functional programming, and get the benefits of emergent design as well?

3 thoughts on “Does math love emergence?

  1. Brian Marick

    I have no idea. However…

    It’s interesting to reflect back on the history of Lisp, functional programming, hackery, particular languages, and Agile. I know only part of the history.

    Lisp was heavily tied into hacker culture at the MIT AI lab and Stanford and Xerox PARC. The Lisp culture of that time was an interesting mix of the academic/mathematical (lambda calculus, the “Lambda the Ultimate” papers) and nitty-gritty (PDP-10 assembler).

    There was a time during Lisp’s heyday when there was a conference called “Lisp and Functional Programming”. At least one prominent Lisp conference organizer is bitter about the Functional Programming people (primarily academic/mathematical at that time) squeezing out the Lisp people. I suspect some part of the answer to your question could come from looking into what happened there.

    It seems to me that the split delayed functional’s emergence into the mainstream, for no good reason. There was too much of a notational/cultural/documentation hurdle.

    Agile culture came pretty strongly out of Smalltalk and OOPSLA-conference culture. I don’t know what intersection that had with Lisp culture (and functional programming culture) – perhaps there was some via Xerox PARC and, secondarily, Stanford? Even though OOPSLA was ostensibly about object-oriented programming, people like Dick Gabriel and Guy Steele–both primarily from the Lisp community at that time–had a strong presence.

    There’s something to be said about how Agile culture was born when researchers and high-end programmers (like Kent Beck and Ward Cunningham out of Tektronix) followed Smalltalk into what were at the time considered jobs beneath them (at places like insurance companies) and applied their brains to situations with ridiculously-quickly-changing requirements. I don’t know enough about the details to say much, but some oral histories would be a good idea.

    What’s struck me, as a bystander to this history, is how *contingent* it is. You (well, I) can really imagine it turning out differently without much change in the world around the people involved.

    So, I guess my answer to your question (“Does math love emergence?”) is that what happens won’t be a matter of any sort of essence of math or emergence — it’ll depend on chance and the people involved.

  2. Ron Jeffries

    I don’t know either. However …

    I happen to have advanced degrees in both math and computer science, and for all that a half-century of software development experience. I mention these solely to suggest that I may have a somewhat integrated view of all this. Then again, I am probably an idiot.

    There is a stark cleanness to the functional style. I think it stems from the notion of immutability. Although in LISP as I learned it, it was possible to edit a list in place, it turns out that when you avoid doing that, nice things happen.

    Most of those nice things come from simplicity, primarily simplicity in one’s mind. If we write pure functions, with no side effects, we just don’t have to think so hard to keep track of what’s going on. Since, no matter how smart we are, they’ll always toss us problems that are right on the edge of too hard for us, keeping this simple is a survival trait. Functional approaches are one way of doing that.

    Even in OO, pure functions turn out to be a good way to go. All the good objects are immutable, and the further we press immutability, the better things seem to go.

    Now to emergence. It is traditional in mathematics to write papers in a particular style. One derives one’s proof or result in a simple straightforward direct way, showing just enough work to convince and just little enough work to show that one has great understanding. Getting the balance right for your audience is difficult: I once got a very poor grade at Michigan for work that at Detroit would have gotten me an A … because at Detroit, all the profs knew that I had the deep knowledge and could talk at their level, while at Michigan, the prof assumed that everyone in the class was a math illiterate and thought that I was hand-waving.

    But the real point is that — unless most people are a whole lot smarter than I am — the derivations and proofs don’t come about in that clean straightforward way. There are scribbles all over the wall and scraps of paper. There are dead ends and backtracking. Things are scratched out with F#%^$! written all over them.

    The proofs and discoveries do emerge. Yes, we may have an intuition that it will be such and so but the work to get there is seldom direct and flawless. It’s just that the papers are written that way.

    Frankly, the same mostly happens in computing. We get to see someone’s final version of their beautiful implementation of … whatever. We do not usually get to see the starts, stops, left and right turns, that it took to get there.

    Understanding always emerges. We have the power, in software development, to use that emergence to deliver value along the way … if we choose to do it.

    I choose to do it, and I wish I had known how to do it sooner. I’ve have won more battles with the business side along the way. Might have even allied with them to create even better products.

    Emergence is what really happens. We can use it. I choose to use it.

  3. rdm

    Personally, I use functional programming primarily because it make experiments, testing and re-architecting fast.

    And, I use agile, because it makes efficient use of my time when developing applications.

    So, how can they possibly conflict?

    Functional programming is indeed a tool for building understanding, and ultimately for any computer system to do anything useful, someone has to understand it and understand how to make those things happen. But it’s also, fundamentally, about computation. If you want to build a UI, you are going to have to deal with non-functional issues.

    And I believe that this principle also holds true for emerging systems. You need to learn new things when building systems that operate in new ways, and functional approaches can be useful there.

    Meanwhile, agile also gives us tools for understanding our systems, but it’s also about dealing effectively with things we do not fully understand.

    So, with that in mind, let’s say I am paired with a functional programmer, and she say “Oh, that’ simple” and proceeds to write a short expression that I do not understand. How should I react?

    In my opinion, the proper reaction would be to inspect the behavior of the code she gave us. I may want to come back later and study that phrase, but that’s a functional reaction and not an agile reaction. For agile purposes, I want to what the thing does, and I want to make sure it does what we need, but it’s not my job to fully understand the details of what goes on inside all the components of the system.

    Though, as an aside, a lot of the tools that I have to work with are not aimed at dealing with phrases of this sort. [Xcode's snippets might be an exception (if they were properly labeled and described), but while you can bring principles you have learned from functional programming into Xcode, the tools you have to work with there seem to be mostly "non-functional". In other most, but not all, of those tools are about building communicating systems and not about building efficient computational architectures.]

    Then again, I also spend a bit of time every day (maybe half an hour) studying such things — functional code written by someone else. And I by “study” I mostly mean taking apart the functions that I care about but which I do not understand, and inspecting the behavior of their components when dealing with data of the sort that I would care about. And, yes, this is a “deviation” from “pure agile” process.

    Likewise, on the functional side, I also avoid “purity”. Purity mostly means not dealing with things outside the scope of that tool. It’s a prim way of expressing the idea: “I only want to have a hammer so I want everything to look like a nail”. And that’s just not how you nurture any sort of emergence of anything.

    Anyways… I suspect that contrasting functional programming and agile programming is like contrasting machine tools with business processes. At an abstract level both are about getting things done in an efficient manner, but at a deeper level they are useful for different classes of problems — though of course there may often be significant overlap where both can be useful when considering specific problems.


Leave a Reply to Ron Jeffries Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>