--- title: "Metaprogramming: Partial Application..." date: 2015-08-26 --- The title of this post looks intimidating. There's a lot of words there that look like they are very complicated and will take a long time to master. In reality, they are really very simple things. Let's start with a mundane example and work our way up to a real-world bit of code. Let's begin with a small story: --- ACMECorp has a world-renowned Python application named Itera that is known for its superb handling of basic mathematic functions. It's so well known and industry proven that it is used in every school and on every home computer. You have just accepted a job there as an intermediate programmer set to do maintenance on it. Naturally, you are very excited to peek under the hood of this mysterious and powerful program and offer your input to make it even better for the next release and its users. Upon getting there, you settle in and look at your ticket queue for the day. A user is complaining that whenever they add `3` and `5`, they get `7` instead of `8`, which is what they expected. Your first step is to go look into the `add3` function and see what it does: ```Python def add1(x): return x + 1 def add2(x): return x + 2 def add3(x): return x + 2 def add4(x): return x + 4 ``` You are aghast. Your company's multi-billion dollar calculator is brought to its knees by a simple copy-paste error. You wonder, "how in Sam Hill are these people making any money???" (The answer, of course, is that they are a big enterprise corporation) You let your boss know about the bad news, you are immediately given any resource in the company that you need to get this mission-critical problem solved for *any input*. Yesterday. Without breaking the API that the rest of the program has hard-coded in. --- Let's look at what is common about all these functions. The `add*` family of functions seems to all be doing one thing consistently: adding one number to another. Let's define a function called `add` that adds any two numbers: ```Python def add(x, y): return x + y ``` This is nice, but it won't work for the task we were given, which is to not break the API. Let's go over what a function is in Python. We can define a function as something that takes some set of Python values and produces some set of Python values: ```haskell PythonFunction :: [PythonValue] -> [PythonValue] ``` We can read this as "a Python function takes a set of Python values and produces a set of Python values". Now we need to define what a Python value actually is. To keep things simple, we're only going to define the following types of values: - `None` -> no value - `Int` -> any whole number (Python calls this `int`) - `Text` -> any string value (Python calls this `str`) - `Function` -> something that takes and produces values Python [itself has a lot more types that any value can be](https://docs.Python.org/3.4/library/stdtypes.html), but for the scope of this blog post, this will do just fine. Now, since a function can return a value and a function is a value, let's see what happens if you return a function: ```python def outer(): def inner(): return "Hello!" return inner ``` And in the repl: ``` >>> type(outer) ``` So `outer` is a function as we expect. It takes `None` (in Python, a function without arguments has `None` for the type of them) and returns a function that takes `None` and that function returns `Text` containing `"Hello!"`. Let's make sure of this: ``` >>> outer()() 'Hello!' >>> type(outer()()) ``` Yay! When nothing is applied to the result of applying nothing to `outer`, it returns the `Text` value `"Hello!"`. We can define the type of `outer` as the following: ```haskell outer :: None -> None -> Text ``` Now, let's use this for addition: ```python # add :: Int -> Int -> Int def add(x): def inner(y): return x + y return inner ``` And in the repl: ``` >>> add(4)(5) 9 ``` A cool feature about this is that now we can dip into something called Partial Application. Partial application lets you apply part of the arguments of a function and you get another function out of it. Let's trace the type of the `inner` function inside the `add` function, as well as the final computation for clarity: ```python # add :: Int -> Int -> Int def add(x): # inner :: Int -> Int def inner(y): return x + y # :: Int return inner ``` Starting from the inside, we can see how the core computation here is `x + y`, which returns an `Int`. Then we can see that `y` is passed in and in the scope also as an `Int`. Then we can also see that `x` is passed in the outermost layer as an int, giving it the type `Int -> Int -> Int`. Since `inner` is a value, and a Python variable can contain any Python value, let's make a function called `increment` using the `add` function: ```python # increment :: Int -> Int increment = add(1) ``` And in the repl: ``` >>> increment(50) 51 ``` `increment` takes the integer given and increases it by 1, it is the same thing as defining: ```python def increment50(): return 51 ``` Or even `51` directly. Now, let's see how we can use this for the `add*` family of function mentioned above: ```python # add :: Int -> Int -> Int def add(x): def inner(y): return x + y return inner # add1 :: Int -> Int add1 = add(1) # add2 :: Int -> Int add2 = add(2) # add3 :: Int -> Int add3 = add(3) # add4 :: Int -> Int add4 = add(4) ``` And all we need to do from here is a few simple tests to prove it will work: ```python if __name__ == "__main__": assert add(1)(1) == 2 # 1 + 1 assert add(1)(2) == add(2)(1) # 1+2 == 2+1 print("all tests passed") ``` ```console $ python addn.py all tests passed ``` Bam. The `add*` family of functions is now a set of partial applications. It is just a set of half-filled out forms. --- You easily mechanically rewrite all of the `add*` family of functions to use the metaprogramming style you learned on your own. Your patch goes in for consideration to the code review team. Meanwhile your teammates are frantically going through every function in the 200,000 line file that defines the `add*` family of functions. They are estimating months of fixing is needed not to mention millions of lines of test code. They are also estimating an additional budget of contractors being brought in to speed all this up. Your code has made all of this unneeded. Your single commit was one of the biggest in company history. Billboards that were red are now beaming a bright green. Your code fixed 5,000 other copy-paste errors that have existed in the product for years. You immediately get a raise and live happily ever after, a master in your craft. --- For fun, let's rewrite the `add` function in Haskell. ```haskell add :: Int -> Int -> Int add x y = x + y ``` And then we can create a partial application with only: ```haskell add1 :: Int -> Int add1 = (add 1) ``` And use it in the repl: ``` Prelude> add1 3 4 ``` Experienced haskellers would probably gawk at this. Because functions are the base data type in Haskell, and partial application means that you can make functions out of functions, we can define `add` as literally the addition operator `(+)`: ```haskell add :: Int -> Int -> Int add = (+) ``` And because operators are just functions, we can further simplify the `add1` function by partially applying the addition operation: ```haskell add1 :: Int -> Int add1 = (+1) ``` And that will give us the same thing. ``` Prelude> let add1 = (+1) Prelude> add1 3 4 ``` --- Now, real world example time. I recently wrote a simple JSON api based off of a lot of data that has been marginally useful to some people. This api has a series of HTTP endpoints that return data about My Little Pony: Friendship is Magic episodes. Its code is [here](https://github.com/Xe/ponyapi) and its endpoint is `http://ponyapi.apps.xeserv.us`. One of the challenges when implementing it was how to avoid a massive amount of copy-pasted code when doing so. I had started with a bunch of functions like: ```python # all_episodes :: IO [Episode] def all_episodes(): r = requests.get(API_ENDPOINT + "/all") if r.status_code != 200: raise Exception("Not found or server error") return r.json()["episodes"] ``` Which was great and all, but there was so much code duplication involved to just get one result for all the endpoints. My first step was to write something that just automated the getting of json from an endpoint in the same way I automated addition above: ```python # _base_get :: Text -> None -> IO (Either Episode [Episode]) def _base_get(endpoint): def doer(): r = requests.get(API_ENDPOINT + endpoint) if r.status_code != 200: raise Exception("Not found or server error") try: return r.json()["episodes"] except: return r.json()["episode"] # all_episodes :: IO [Episode] all_episodes = _base_get("/all") ``` Where `_base_get` returned the function that satisfied the request. This didn't end up working so well with the endpoints [that take parameters](https://github.com/Xe/ponyapi#seasonnumber), so I had to account for that in my code: ```python # _base_get :: Text -> Maybe [Text] -> (Maybe [Text] -> IO (Either Episode [Episode])) # _base_get takes a text, a splatted list of texts and returns a function such that # the function takes a splatted list of texts and returns either an Episode or # a list of Episode as an IO action. def _base_get(endpoint, *fragments): def doer(*args): r = None assert len(fragments) == len(args) if len(fragments) == 0: r = requests.get(API_ENDPOINT + endpoint) else: url = API_ENDPOINT + endpoint for i in range(len(fragments)): url = url + "/" + fragments[i] + "/" + str(args[i]) r = requests.get(url) if r.status_code != 200: raise Exception("Not found or server error") try: return r.json()["episodes"] except: return r.json()["episode"] return doer # all_episodes :: IO [Episode] all_episodes = _base_get("/all") # newest :: IO Episode newest = _base_get("/newest") # last_aired :: IO Episode last_aired = _base_get("/last_aired") # random :: IO Episode random = _base_get("/random") # get_season :: Int -> IO [Episode] get_season = _base_get("", "season") # get_episode :: Int -> Int -> IO Episode get_episode = _base_get("", "season", "episode") ``` And that was it, save the `/search` route, which was acceptable to implement by hand: ```python # search :: Text -> IO [Episode] def search(query): params = {"q": query} r = requests.get(API_ENDPOINT + "/search", params=params) if r.status_code != 200: raise Exception("Not found or server error") return r.json()["episodes"] ``` --- Months later you have been promoted as high as you can go. You've been teaching the other engineers at ACMECorp metaprogramming and even convinced management to let the next big project be in Haskell. You are set for life. You have won. --- For comments on this article, please feel free to email me, poke me in `#geek` on `irc.ponychat.net` (my nick is Xena), or leave thoughts at one of the below places this article has been posted. Comments: - [Hacker News](https://news.ycombinator.com/item?id=10127389) - [Reddit](https://www.reddit.com/r/programming/comments/3ijyyz/metaprogramming_partial_application_and_currying/)