jump to navigation

Begin with the Ending July 26, 2016

Posted by PythonGuy in Uncategorized.
add a comment

I don’t know what it is, but I’m working on a team now where some people like to write a bunch of code, test it, and then integrate. To me, that feels backwards.

My preferred order is: write a tiny bit of code, integrate that, and then write a lot of code, testing the integration along the way. Unit tests and such often come last, and then only when I’m not able to easily test with integration tests.

The way you do this is through scaffolding. Say you have a client and a server. I would write a minimal server. Then I would write a minimal client. Then I would have the client call the server. After that, members of my team can start working on different features, which means modifying the client and the server to provide that feature.

Writing code this way isn’t a license to be stupid. You still have to think hard about how you want things to work. It does, however, free you from a lot of tedious and unnecessary details when planning. IE, you should be focusing on what messages are passed back and forth, and the general content of those messages, not specific parameters and fields.

Notes on Pip July 26, 2016

Posted by PythonGuy in Uncategorized.
add a comment

Why pip?

Pip manages your python code rather well. It downloads and installs dependencies and makes your python experience almost seamless.

Pip combined with virtualenv is a powerful tool. In fact, I never use the pip installed at /usr/bin/pip/. I always set up a virtualenv and use the pip installed there.

Installing 3rd Party Packages

Pip can take URLs but it usually takes package names of packages on PyPi. You can even specify versions.

Here’s how to do that:

pip install <package name>

If you want a specific version:

pip install <package name>==<version number>

You can install multiple packages at once, too.

But note, you really shouldn’t be installing more than one package. If you’re developing your own package, you shouldn’t be installing anything but your own package.

Installing Your Own Packages

Create a setup.py and layout your code properly. I won’t document that here; you can look elsewhere for instructions. Then run pip to install your package. It will install all your dependencies as well.

pip install -e <directory with setup.py>

If you want to install the test or dev dependencies:

pip install -e <directory with setup.py>[test]
pip install -e <directory with setup.py>[dev]

NOTE: If you are using zsh, you have to quote things properly.

The -e flag puts the files in as symlinks to the original code files. You can modify the files in your project and they will be modified in the installed location.

Bad Practices

  • Not writing a setup.py because you think your code is not a package. All the code you write should be a package.
  • Doing pip freeze > requirements.txt. Just use setup.py and list your dependencies explicitly.
  • Not using virtualenv or similar for each project.


Things in ViM I Really Miss June 22, 2016

Posted by PythonGuy in Uncategorized.
1 comment so far

When I am using a different text editor or (blegh! a word processor), there are certain things I miss. Namely:

  • quickly searching for things with /<regex>
  • quickly replacing with :%s/…/…/
  • marking points and jumping between them.
  • Jumping to the end of a sentence or the end of a paragraph or the end of a word, or the beginnings of the above.
  • Repeating the last command I just did.
  • Recording a macro and running it multiple times.

I really wonder how you heathens who never learned an advanced text editor like ViM get along with your life. It’s like watching you trying to create a fire with a stick and a string while I have a microwave.

GPL Thoughts May 11, 2016

Posted by PythonGuy in Licensing, Uncategorized.
Tags: , , , , , ,
add a comment

It’s 2016, and the GPL is once again on my mind. Broadly speaking, there are three types of licenses out there.

  • Free Software, which is software licensed such that it will always remain freely available and open to modification. As a side-effect, it will “infect” other software that uses it such that it becomes free as well.
  • Open-source Software, which is software licensed such that it can be freely shared or modified, but it doesn’t “infect” other software.
  • Proprietary Software, which is software licensed such that it cannot be freely shared or modified.

There is a fourth type, but it barely deserves mentioning. “Public domain” software, which is software that is practically dead.

When Stallman came up with the GPL, his interest was to fundamentally change the way we write, use, modify and share software. In order to accomplish this, he set into motion the following plan:

  1. Write some good software and freely share it, but make sure it cannot be part of any proprietary software.
  2. Write more good software that competes against and replaces proprietary software.
  3. Eventually, it won’t make sense to write proprietary software anymore, because all the good software is free and nobody expects to pay for software.

According to Stallman’s plan, we’re very deep into 2 and well on the way to 3.

The question that inevitably arises, “How do developers get paid?” is forefront on my mind. After all, I write code so that I can get paid. (I would probably write code otherwise, but not nearly so much.)

So how can I get paid for my work?

Truth be told, I make more money from supporting software than anything else. It’s very rare that I get to write entirely new software, even when I’m working on a software project. Most of the time, I am fixing already existing code, or adapting it to some purpose, or more likely, testing it to see if it actually does what we think it does, or just to figure out why it doesn’t do what I want it to do.

The tools that I use are almost all GPL. Or rather, if all of my tools were GPL it wouldn’t hurt me in the slightest. In fact, it would make my life a lot better.

But then how do I make money?

Simply put, people still want software to be written, and it isn’t unheard of to have people pay me to work on open source projects. It isn’t unthinkable that I could be hired to work on free software as well, if there was a great need for that. There are not a few developers who are already being paid by private companies to do that.

The big question then isn’t, “How do individual developers get paid”, but “How do we convince people to pay for software development work?” The answer is when there is a need, people will pay. Sometimes very large sums of money.

The proprietary model doesn’t make any sense anymore. Proprietary software is an agreement that looks like this. “Give me your money, I’ll take it and I’ll give you software that you can’t look at or modify. If you like the software, keep giving me money. If you want to make the software better, beg me to do it for you.” That just doesn’t fly.

I think we’ll see companies that arise that hand out their software for free. In exchange, they will get paid to modify the software or to teach people how to use it. They may also get paid to adapt the software for a particular environment. Thus, the value in a software company will not be the software, but the developers and the ability of the company to apply the software to solve problems.

This, to me, is tremendously encouraging. If this vision comes to pass, I won’t be hired to write software and fired when I finish. I’ll be hired to staff companies so they can say, “Whatever you need done, we can do it, provided you can afford our developers.”

The GPL really is the way forward, and free software is the only good solution out there.




Shared Memory and Python May 10, 2016

Posted by PythonGuy in Advanced Python, GIL.
add a comment

I’m researching shared memory and Python right now, because it seems like the only hope for a particular situation we are seeing.

Basically, we have a very high-performance web server that is designed to handle a large number of requests per second. At the same time, we want to be able to update the code that is used to process these requests, real time.

Our webserver is using microthreads (greenlets) and so it can only really do one thing at a time, even though it looks like it is doing many things. When we go to update the code, everything else must stop until the update is complete.

Obviously, we can use rolling deployments, but there are some particular issues we see with that. Namely, memory and resource management.

If we did “hard” rolling updates, we would have to turn off a node, taking it out of service. Then we’d perform the operation, and then add the node back to the service. This would take a non-trivial amount of time.

If we did “soft” rolling updates, we would spin up a new web server on a node, flip from the old to the new server, and retire the old server. This requires twice as much memory as we typically need.

Another option would involve shared memory. The code would be served from shared memory. When we’d like to do an update, we’d launch another process to create a new chunk of code in shared memory, then we’d flip from the old to the new code. This seems like the ideal option.

The problem is that Python only supports the most basic data structures in shared memory.

Perhaps there is a way to trick Python into treating shared memory as actual Python code, maybe by storing the code as an array. When we go to run the code, we load the code from the array directly into a function and then call it. I don’t know if that would work, or that any method would work, but I’m looking into it.

Suggestions welcome!

The Pastiche Test May 2, 2016

Posted by PythonGuy in Uncategorized.
1 comment so far

I found this article fascinating: The pastiche test

The idea is sound, and I wish I knew what it was called before he named it, if it had a name.

The idea is this: Can you implement core language features in the language itself? That is, can you write something that would do the same thing as “if” using functions or whatnot?

There have been more than one occasion where I would have like to have had access to, or be able to replicated, some of the core features of Python. If Python completely passes the pastiche test (and PyPy is proof that it can), then that would not have been a hurdle.

Lazy Evaluation by Default May 2, 2016

Posted by PythonGuy in Uncategorized.
add a comment

I’ve been thinking a lot about a programming language idea I had where function calls do not execute the parameters before hand. It sounds interesting enough but I think it’s not anything new. It’s a style of programming called “Lazy evaluation”.

It works like this.

Normally, you’d have code that looks like this:

a = 1 + 1

b = a + 1

and you’d expect the following operations to occur:

  1. Add 1 and 1 (2)
  2. Assign 2 to a
  3. Add 2 and 1 (3)
  4. Assign 3 to b

With lazy evaluation, none of the above happens. Instead, this is what happens.

  1. Assign a to “1 + 1”
  2. Assign b to “a + 1”

Note that none of the addition operations are performed. Instead, if you wanted to get the value for b, you’d have to tell it, “Calculate your value.” In which case, the following sequence of events would occur:

  1. Calculate a.
  2. Calculate 1 + 1
  3. Calculate 2 + 1

Note that this demonstrates how you can nest lazy evaluation.

A beautiful aspect of lazy evaluation is that values that are never used are never evaluated. This can be a problem when you consider functions that have side effects. If there is a value that depends on the results of a function with side effects, that function will never get called and those side effects never manifest unless the value that depends on the function is evaluated.

But this can actually be a benefit. Remember how when you do “a or b” on Python it doesn’t evaluate b if a is True? This makes the “or” and “and” operators behave something like “if”. That is, don’t execute an entire branch of code if you’re not supposed to.

Just a thought at this point. I might try writing some pseudo-code to see how it all works out.

Expose the Internals April 29, 2016

Posted by PythonGuy in Uncategorized.
add a comment

I think it would be cool if Python exposed some of its inward functionality. These are some of the bits I would like to have:

  • The function that maps function arguments to parameters. It would take a function parameter list, an argument list and an argument dict and convert it to a dict of variable-value mappings.
  • The function that creates functions. I know I can use exec for this, but I think it would be preferable to allow me to specify a string and get back a chunk of un-executed code.
  • The lexer and parser. Along with this, I’d like to be able to create arbitrary data structures representing the output of the parser and put that into exec.

I don’t know how much work it would be to expose these parts. I should look into making a module that does this. It can’t be that hard.

More on my ideal language April 29, 2016

Posted by PythonGuy in Uncategorized.

Consider the following possibility:

  1. All statements and expressions are not evaluated by default. That is, “1+1” does not mean “2”, but literally, “add 1 and 1.” When I set “a = 1 + 1”, I am really setting “a” to be “add 1 and 1”. Or rather, it literally means “set a to 1+1”. It doesn’t actually do anything.
  2. There is a language feature to execute statements. I think they use “`” in Lisp. I would rather have it a function just like any other, though. I’d call it “exec”. So, “a = 1+1; exec(a)” would give me 2. But do you see the problem? “exec(a)” literally means “exec(a)”, which does not execute it but describes the action of executing it. So perhaps “exec” is special in that it is the only function that actually executes anything.
  3. With such a feature in place, it is simple to see how to make the if statement into an if function. IE, “if(x<y, return(7), return(5))”. This function would take three parameters, none of them executed until if() said so. It would execute the first parameter, and then choose between executing the second or third.

I think the above features would give me the equivalent of Lisp’s macros in a Python-like syntax. It would be interesting to try and program completely in such a manner, though. I would like to invent a language just to see what it would be like.

Verbs don’t need Nouns April 29, 2016

Posted by PythonGuy in Uncategorized.

A shower thought, more or less, about verbs, nouns, and the nature of computer programming languages.

Verbs can live on their own. They don’t need nouns. We see this because in many languages that require nouns, we use placeholder nouns. IE, “It is raining.” What is “it”? It is literally nothing. We are describing the state of the universe. We could’ve dropped the subject altogether and lost no meaning at all: “Is raining.” That’s almost perfectly clear, even in a language like English where every sentence needs a subject noun.

Nouns, however, cannot work alone. They need a verb to tie it into the universe somehow. For instance, “Baby!” means, “There is a baby!” The verb, “is”, is implied, and is part of the idea being expressed. Or, in a auto repair shop, “7 mm” would mean, “Please hand me the 7 mm wrench.”

I think you’ll find that as you look at the fundamental ideas in language, verbs will always be the most important. Nouns alone can only express phrases or parts of ideas, not complete, whole, independent ideas. Sentences are the basic building blocks of any language, and every sentence needs a verb, even if it is the lowly connecting verb or the lowly declaration-of-existence verb.

In programming languages, we write different kinds of statements. These statements generally take one of the following form:

  1. Declarations. There is this thing called X and this is what it is like. In Python, class and function declarations fall into this category.
  2. Procedure descriptions. The fundamental unit here is a single unit of action. Assign this value to that variable, call this function. All procedure descriptions can be broken down into more basic procedure descriptions, all the way down to the fundamental operations the language provides. In this way, the operators and basic functions form the core of the language that all the rest of the language relies on.

Honestly, it’s very hard to distinguish between the above two types of statements, suggesting there may be some more fundamental concept. After all, a class statement is really saying, “Create a class object given these parameters and assign it to that variable which happens to be the same as the class name.” The same goes for function declarations.

In the end, I believe the fundamental unit of any language, even Java, is the function. All languages are really a subset of Lisp or Scheme.

Viewing languages this way is certainly liberating. It’s hard to explain why, but when you see that all things are really the same thing done in different ways, your mind has an easier time inventing new things.


Get every new post delivered to your Inbox.