The way it works is you have a population, an amount of land, and an amount of grain. Every year, you need to choose how much grain to plant, how much grain to feed the people, and how many acres of land to buy or sell.

Each year, a certain amount of grain would grow, people would starve if they didn’t get enough food, and rats would eat your surplus grain. Random events would cause disasters to make things interesting.

The game was remarkably simple but surprisingly addictive.

I figured it was time to write a new version for the modern age, when memory and CPU are no longer real constraints to most of the problems we faced programming 30 years ago. I chose the Hamurabi game as inspiration.

Of course, a real simulation of a primitive society could track each individual within the population. People would age, and then die from starvation, random events, or just old age.

Using the US Census data, I found some actuarial tables that gave a percentage chance someone would die at a particular age.

I stored the number of people in each age group, separated by sex, in a simple array. The index was the age.

Now all I needed to do was find a scalable way of calculating how many people die in each age group every year.

The simplest way to do this is to roll the dice for each person each year. Simply use the random modules’ random() function to get a number between 0.0 and 1.0. If this is left than the probability they will die, then they die.

Of course, if you have a thousand people, you have to call random() a thousand times. A million people need it called a million times. This would slow the program down as your population grew.

I vaguely recalled some math from my college years that should help me work these things out more quickly. Reading some of my old textbooks, I discovered that that Binomial Distribution gives you the exact probability of seeing x events in population of n where each has a probability of p.

However, the binomial distribution relies on combinatorials, which rely on factorials, which are notoriously difficult to calculate precisely for large numbers.

Luckily, this is not a new problem. Approximating the Binomial Distribution leads to the Normal or Gauss Distribution. This relies on simple exponentials and Python’s random module already has a gauss() and a normalvariate() built in.

What do you use for mu and sigma, the mean and standard deviation? The mean is simply n*p, and the standard deviation is the square root of n*p*(1-p).

The normal distribution really doesn’t give you good approximations when n or n*p is small. If n is really small (there are only a handful of people), I can just roll random() that many times. But if n is large, and p is very small (generally, for younger ages), then my approximation breaks down.

For this, the Poisson Distribution is useful. The Poisson Distribution can give you a very good approximation of the Binomial Distribution for small numbers of deaths. What I do is roll the dice with random(), and walk from 0 deaths upward, subtracting the probability that that many deaths would occur given the Poisson Distribution. When I’ve exceeded the roll, then that’s the number of deaths that occur.

Using these three methods of calculating deaths lead to a very scalable algorithm for calculating the number of deaths in a population of 1, 10, hundreds of even millions of people. I ran my simulation for several hundred years until I saw hundreds of billions of people, with tens of billions dying each year.

I think this shows why I like Python. Never once did I have to think very hard about how to express my ideas in Python. I was completely free to consider the algorithm. Most of my time was spent reading math books and translating complicated formulas into simple algorithms, and testing that it did what I expected.

I ran into an issue today that demonstrated why this is a good philosophy.

In the abstract view, I had a system with several individual component processes. Each process depended on the output of the previous process, through a roundabout way. (One accepted internet connections, another accepted log messages in our log format, another analyzed those log messages, and another summarized the analyses.)

We ran into a problem. A less-than-honest user of our system fed some very bad data in through the internet connection. That process didn’t particularly care what the input looked like, and fed it on through the rest of the system. It wasn’t until much later than an exotic problem arose that shut the entire system down.

As we meddled with the backend process that barfed on the bad data, we decided that that kind of data wasn’t valuable in the first place, so we might as well shut it down from the beginning.

If we had only implemented that fix, then our system would still be broken. See, we still had bad data in our system that had to be processed. So the dual fix of (a) eliminated the acceptance of obviously bad data, and (b) building a system more tolerant of bad data were needed.

I am often amazed by the engineers in the physical world. These people build machines that have the same issues with bad input, except their bad input includes “dust in the air” and “metal filings that are created by wear and tear.” They have to build even the most sensitive components of their system to be tolerant of this kind of bad data, even though they install filters and covers and such to eliminate as much of it as possible. If they didn’t do so, then the entire machine would come screeching to a halt the first time something bad happened.

We need to build our software the same way. No matter how internal a component is, it needs to be ready to accept bad data.

I am poking at a long function, more than my 1080p rotated monitor can fit on one screen at a readable font size for me. It flows off the edge of the screen, making it hard to see how different pieced fit together.

I don’t like long functions. When you reach a certain number of variables in a particular scope, or when you have bits and pieces of your code spread out like peanut butter, it becomes unreadable.

Of course, long functions are probably better than the alternatives, provided you can’t separate out different parts of it.

The author of the code I am reading decided it wasn’t a good idea to put relevant pieces of code together. As a result, the code is intertwined.

Variables are created in one spot and used in another. In between, they are not relevant at all.

  the_var = the_value

  ... some irrelevant bits of code ...

  use(the_var)

Obviously, it is much more readable if you do things this way:

  ... irrelevant bits of code ...

  the_var = the_value
  use(the_var)

  ... other irrelevant bits of code ...

Or even better, eliminate the variable altogether:

   use(the_value)

By minimizing the mixing of code, it becomes much easier to read the code and put things together in my head. This is no different than writing English in paragraphs, where each paragraph has sentences that relate to each other.

In addition, you can make your code more readable by sensibly keeping the variable count down. Sometimes, reducing the number of variables means you have a more complicated expression, so you have to think carefully about how to do this.

Comments, in addition to vertical whitespace, can be used to separate “paragraphs” of code. I use single-line comments when I’m talking about the next few lines, but multi-line comments when I’m talking about entire sections of code.

#
# This is a description of the following block of
# code. Note how it sticks out because of the white space.
#

# This is a comment about the next two lines of code.
do_something()
while you_do_something_else():
   append_this_to_that()

# This comment applies to the next few lines.
this = that + that_other_thing - 5
these = [that, that_other_thing]

I hope my point is clear. You can write wonderful code, but make it entirely illegible, even in Python. So don’t do that. Take a little bit of time to make your code readable. Your future self will thank you.

Before we begin, let’s remind ourselves that the Python pattern for sequences is to include the first item, but exclude the last. That’s how range() and all the other functions work.

I’ve seen people often do something like this:

days = (some list-generating code)
for day in days:
   ... process that day ...

This may be how you were trained to do this in other languages. I strongly recommend not doing this way. In general, using lists when you don’t really need them means you’re passing up some wonderful features of Python that will make your code more readable, more robust, and faster.

Some languages encourage you to avoid lists and use the 3-clause for loop that is not found in Python. I prefer lists over 3-clause for loops, mostly because I’ve never met one that was clear and obvious in meaning compared to the iterative approach.

For finite sequences of days, I prefer the generator. It gives me the highest chance of getting things right from the beginning.

for day in (... generator for a sequence of days ...):
    ... process that day ...

What is appropriate for that generator? A generic generator function should do the trick:

import datetime
def generate_days(start_day, end_day):
    day = start_day
    while day < end_day:
        yield day
        day += datetime.timedelta(days=1)

The above pattern is so common it’s a wonder why we don’t generalize xrange. The interface could be something like:

xrange(first=0, last, increment=1)
Returns first, first+increment, first+increment+increment, etc until but not including last. Requires that increment can be added to first and that the current item can be compared with last.

One can dream…

You’ll note that if we write our own general xrange generator, it is just this:

def our_xrange(first, last, step):
    cur = first
    while cur < last:
        yield cur
        cur += step

Of course, using itertools, you can combine the count() and takewhile(), but count() seems to only care for numbers. It’s trivial to write your own count() generator, though.

You’ll notice that I’m not dealing with numbers and then converting those numbers to dates. I much prefer objects that know what month and day they are in to integers that require lots of math to massage their meaning out of them.

Finally, I want to mention one special case that arises with databases. Oftentimes, you want to query data from the database for a sequence of dates. While some databases provide advanced features to allow you to do this within the database, many (most, now that NoSQL databases seem to be common) do not. I wouldn’t rely on the database doing the hard work for you. Besides, as I’ll show below, it’s really not that hard in Python, probably much easier than the solution the database provides.

There are two ways to handle this. One, we could pre-generate a list of dates, and then query the database for each date with an “in” condition (SQL). Two, we could ask for all the data between two date ranges, sort them, and then walk through the data in parallel with our date counter in Python.

The first is not desirable. This is because you have to send a lot of data across the wire—one date for each point on your graph, or row in your table. Let’s ignore it and move on.

The second is more desirable, but the problem of iterating through two lists in parallel is not something that’s obvious to most programmers. There is a simple solution, but it takes a bit of explaining. I’ll try to summarize that here.

What you want to do is to first generate the iterator that will give you all the dates you are interested in, given only the start, end, and interval. Then you want to query your database for all points of data that align with this, giving only the start, end, and interval. You’ll need to sort the data coming from the database. This will end up in another iterator of some sort.

Next, you want to iterate across your date iterator, performing some action. You’ll want to grab the row from the database only if the next row matches that date.

Here’s some sample code.

def date_result_pairs(dates, results):
    """Generates pairs of (date, result), one for each date in dates. If there is no corresponding result, then the result will be None."""
    dates = iter(dates)
    results = iter(results)
    try:
        result = results.next()
        for day in dates:
            if result.date > day:
                yield (day, none)
            else:
                yield (day, result)
                result = results.next()
    except StopIteration:
        for day in dates:
            yield (day, None)

dates = our_xrange(start, end, interval)
results = (query the DB for all data between start, end, matching interval, sorted by date.)
for date, result in date_result_pairs(dates, results):
    print "Result for date %s is %r" % (date, result)

This isn’t perfect. It assumes the database is returning dates that match exactly with the dates you have, which may or may not be correct given your ability to write the correct query.

I hope this has demonstrated some of the elegance of Python for these kinds of problems.

As always, questions and ideas are welcome.

There are, in general, two areas of blame:

• Poor implementation of query code
• Poor implementation of database schema

Granted, SQLAlchemy makes things look deceptively simple. It handles so many optimizations for you that you generally should go with the first idea that pops into your head. You can get a lot done with bad code.

However, at some point, you need to think, really hard, about what you want to get down when and how.

I won’t even touch on the topics of optimizing your database. There are innumerable resources out there to discuss when and when not to use indexes and how to arrange your data for the best query response.

I will say, however, that you need to think about which machine is going to do the work that needs to be done. Is your database going to do all the number crunching, or are you going to send the data to your app and have it do the number crunching? There are arguments for both ways.

The database is going to have indexes and the most recent data for you. If you try to limit the number of queries you make and the amount of data you send back and forth between the database, you’re probably doing things right.

Occasionally, the operation is better handled by having the database send the data to the app and the app works it over. These cases are rarer than the former.

With that in mind, let’s look at some simple optimizations to push the work on to the database and outside of your app.

Optimization 1: Are you making too many queries?

A good hint that you are making too many queries is when you put a query inside of a for loop iterating over the results of another query. In general, these queries can be combined. It will require, however, that you learn a little bit more about SQL and how to write those SQL statements in SQLAlchemy. You may want to visit the advanced topics of nested SELECT statements, and fully understand it, so that you can see how to optimize your nested queries.

Another problem I see from time are hidden queries. Look at your logs. Identify which queries match which lines of code. If you can’t point to where the query originated from, then you’ve got a hidden query. You need to understand more fully what exactly your SQLAlchemy model is and which attributes are spawning which queries.

Optimization 2: Are you gathering too much data?

The obvious case is when you have big tables, or tables that join with many tables, and you are sucking down more data from the database than you want or need.

The way to resolve this is with a healthy dose of deferred() and grouping of those deferreds.

Spreading your data across multiple tables makes this a bit clearer. For instance, do not put your metrics data in the same table as the attributes of the object. Seldom do you want both the color of something and the number of times someone bought it at the same time.

Optimization 3: Pagination.

This one is so obvious for an experienced SQLAlchemy user, but is unfamiliar to novices.

SQLAlchemy provides powerful mechanisms for pagination. When you want to do pagination, follow this simple formula.

Specify the query for the data you want, including only the columns you are interested in. Be sure to join with all the tables you need to join with in order to get the data you need. (NOTE: You can specify columns as the parameters to the query() method of your Session!) Be sure to include whatever sorting or filtering you need as well.
Run the count() on that query. This is the total number of rows. Certain databases do not like the count() call very much, so if you need to, store this someplace safe and temporary, such as a cookie or session or even in the HTML form.
Append the modifiers limit() and offset(). Some databases do not respect these, but most of them do.

Optimization 4: Sanity.

As a final check, just poke around the logs of both your app and your database and see how long each of the queries take. Identify the slower, more common ones, and attack those.

Are the queries necessary? Do they gather too much information? Or should they be paginated?

Finally, you’ll want to revisit your schema and indexes to see if there is a better way for the database to handle that data.

Optimization is an ongoing process. Try not to get hung up on premature optimization, and try to not let your unfamiliarity with SQLAlchemy or your database limit you. Take the time to learn the more advanced features of both, so that you can apply the full power to your problems.

I started with the simplest case. I had three Numpy arrays. One was an array of vertices, another an array of normals, and the third an array of triangles which referenced three vertices and three normals.

Rendering the surface in OpenGL is trivial. Just glBegin(GL_TRIANGLES), then go through each triangle, calling glNormal3dv() and glVertex3dv() for each normal and vertex in the triangle.

Thinking of this in the computer science sense, it’s apparent that the list of vertices and the list of normals are not truly independent. If a vertex or normal isn’t mentioned as part of a triangle, there’s no reason to hold on to it in memory.

In a sense, the arrays of vertices and normals is an artificial data structure, created for convenience, not correctness. If I manipulated the surface so that triangles would change or disappear, there’s a good chance that the arrays of vertices and normals would be incorrect, holding information that is no longer needed.

I started down the OO path, creating a Triangle class, then a Vertex and Normal class. I realized, part way through this process, that I was moving away from the way things work in OpenGL land. Ultimately, I would load the vertices and normals into a buffer of some sort, and then I would issue render commands against the buffer of vertices. I haven’t quite seen how to do this yet, but it’s apparent that the array is the proper way to store these things, not as separate objects spread out all over creation.

The second problem was one of elegance. In the system I had devised earlier, it is quite natural that many triangles would share the same vertex, and these relationships are all but obvious. If I moved the vertex, then all of the connected triangles would likewise move, whether or not they were aware of the change. While I can implement a system where the same Vertex object is shared among many Triangle objects, I doubt it would be as elegant as the first solution I derived. Finding which triangles share the same vertex is not trivial in the OO approach, but in the indexed approach it is a simple filter for triangles that have the same index to the vertex.

I haven’t exactly resolved this dilemma in my mind. I suppose that I can create an object that uses invisible indexing into a massive array, and manages the memory within the array not unlike you’d have to do in C. I’m not convinced that such a solution is elegant or even beneficial.

In the long run, I think having a highly optimized set of data structures, each with their own unique access and modification rules, is probably the right way to go. I can “protect” these within a class, and then provide accessors and mutators to view and modify specific segments as needed. Perhaps having the elements defined as low as individual Triangles and Vertexes is simply too much detail.

I have seen a lot of Python code, as well as Java, Perl, and other languages. When dealing with dicts, I have identified a few patterns that I feel are optimal, depending on the situation. These are extraordinarily simple.

NOTE: The dict data structure is known by many names: map, a hashmap, an associative array, a hash, or even a table.

Use if Present

For instance, how many times have you written the following pseudo-code in your language of choice?

# pseudo-code
if key is in map:
    lookup value in map with key
    use value

Python provides the “get()” method which returns None if it is not present.

# Python
value = map.get(key)
if value is not None:
    # use value

Of course, sometimes you need to distinguish between values that are None and that are not present in the dict. You can rely on exceptional behavior for this:

# Python
try:
    value = map[key]
except KeyError:
    pass
else:
    # use value

Or you can use the “in” test:

# Python
if key in map:
    value = map[key]
    # use value

The above, of course, does 2 dict lookups.

Exceptions versus Lookups

There is discussion about whether to use exceptions or lookups. The general rule of thumb is that exceptions are not as slow as you think, since Python is pretty slow to begin with. “Slow”, of course, is a relative term that is meaningful when you compare Python to C/C++. And nowadays with PyPy, “slow” isn’t a proper word for it anymore.

Use Value or Default If Missing

Sometimes you want to use a default value if the key is not present. This is simply:

# Python
value = map.get(key, 'default')

Note that whatever expression you use as the default value will be evaluated, whether or not it was used. If the expression is expensive to calculate, then you can use this form:

# Python
value = map.get(key)
if value is None:
    value = expensive_expression()

Notice that you’re back to the previous pattern if you need to distinguish between a value of None and a missing key.

Use Value or Default and Store if Missing

Sometimes you want to store the default value in the dict if it is missing. “setdefault()” is the ideal method for this.

# Python
value = map.setdefault(key, 'default')

Of course, the caveats for expensive default expressions applies.

Conclusion and Summary

Those of you who are unfamiliar with Python might note how similar all of the above patterns are. Indeed, if you simply learn what the following expressions mean, you don’t have to think very hard to understand what the code does or to choose the right code:

• key in dict
• dict[key]
• dict.get(key)
• dict.get(key, default)
• dict.setdefault(key, default)

If you want to follow along, I’m going to try and build a Minecraft clone.

The general idea of the game is that you control a character that can manipulate the world. I’m going for a medieval RPG style game play, maybe with some networking enabled in the future.

The problem of generating the cubes is harder than it looks. My big idea is to apply a smoothing algorithm so that the data looks more natural. That is, if two blocks only differ in elevation by 1 unit, then the two blocks should form a neat slope.

Another idea is just to start with raw data describing the landscape, and allow the player to manipulate the mesh.

Anyway, OpenGL is very well documented, both in the infamous Red Book and on the OpenGL wiki. I encourage you to check it out.

PyOpenGL, by the way, is the way to go.

I feel their pain, but not the pain of embedding Python. I feel their pain of being forced to write code in C.

As Glyph Lefkowitz so clearly explains, the fact that Lua is easily embedded doesn’t make it a good choice. Embedding is the wrong engineering choice to begin with.

What you should be doing as a programmer is creating usable code fragments. If the Free Software movement has taught us anything, the survivability of code depends on its usefulness, and that usefulness goes into how other programs can tap into its power.

If you’re writing an environment from which you expect other programmers will confine themselves within, you are making a big mistake. Those programmers who fall into this trap will one day realize that they’ve wasted their lives in pursuit of the impossible. If you want to know where things will go, go read about Emacs and Lisp. That’s what you’re recreating.

If, instead, you’re writing components of programming intelligence, buried into .so’s or .dll’s, then other programmers can follow after you and incorporate your program into theirs.

Lua might be a nice embedded language, but embedded languages are not what you really want.

I agree and concede the point that Lua’s language is smaller. I also admit that Lua’s code base is much, much smaller.

However, does this translate to simplicity for the programming task?

I think Lua is wonderfully engineered. The guts of Lua are beautifully assembled. It is a wonderful structure to behold, much like the Taj Mahal.

However, people don’t live in the Taj Mahal. They live in modern homes which are an engineer’s nightmare. Modern homes are much more comfortable because the modern conveniences that make living simple are readily available.

Behind the faucet is miles of piping. Behind the outlet is an electrical system that will confuse even the most experienced electrician. But we know that if we want to live, we would rather live in a modern home than the Taj Mahal.

Yes, the Taj Mahal is simpler, being constructed out of stones that fit perfectly together. Yes, we can appreciate how a few simple components have come together to make something greater than the whole. But that’s about all we can do.

A very, very good example is how Python approaches Object-Oriented Programming versus Lua’s approach to Object-Oriented Programming.

Python provides not insignificant language constructs, embedded deep within the syntax and interpretation, that make simple classes trivial to create, and hard classes easy.

Python’s OO system is easily learned. When you need special behavior, such as attribute look up and assignment magic, the way to do so is plain and easy.

Lua’s OO system, on the other hand, is extremely limited. You only have two or three syntactic elements to support OO programming, and these are not “magic”, nor can the magic be modified to fit the task.

Where is SQLAlchemy in Lua? It cannot exist. For Lua’s “simple” system, you will never have declarative ORM, and never understand what it means to have a class backed by a database. Sure, SQLAlchemy is much more complicated than anything you would ever dream of writing in Lua, but Python has provided the modern conveniences that not only make writing SQLAlchemy possible, but almost easy.

Why does Python have so many modules? Because it is so easy to create new modules that do interesting things, whereas in Lua, this is simply not done.

Yes, Lua, the language, is simpler. But I don’t want to use a language that is simpler. I want a language that makes my job, the task of programming, easier. I don’t care how complex that language is, anymore than I care whether how complicated the grammar in English is. I simply care how hard it is to do my job, and Lua makes my job harder, not easier.