431 lines
11 KiB
Markdown
431 lines
11 KiB
Markdown
---
|
|
title: "Metaprogramming: Partial Application..."
|
|
date: 2015-08-26
|
|
---
|
|
|
|
Metaprogramming: Partial Application and Currying 101
|
|
=====================================================
|
|
|
|
The title of this post looks intimidating. There's a lot of words there that
|
|
look like they are very complicated and will take a long time to master. In
|
|
reality, they are really very simple things. Let's start with a mundane example
|
|
and work our way up to a real-world bit of code. Let's begin with a small
|
|
story:
|
|
|
|
---
|
|
|
|
ACMECorp has a world-renowned Python application named Itera that is known for
|
|
its superb handling of basic mathematic functions. It's so well known and
|
|
industry proven that it is used in every school and on every home computer. You
|
|
have just accepted a job there as an intermediate programmer set to do
|
|
maintenance on it. Naturally, you are very excited to peek under the hood of
|
|
this mysterious and powerful program and offer your input to make it even
|
|
better for the next release and its users.
|
|
|
|
Upon getting there, you settle in and look at your ticket queue for the day.
|
|
A user is complaining that whenever they add `3` and `5`, they get `7` instead
|
|
of `8`, which is what they expected. Your first step is to go look into the
|
|
`add3` function and see what it does:
|
|
|
|
```Python
|
|
def add1(x):
|
|
return x + 1
|
|
|
|
def add2(x):
|
|
return x + 2
|
|
|
|
def add3(x):
|
|
return x + 2
|
|
|
|
def add4(x):
|
|
return x + 4
|
|
```
|
|
|
|
You are aghast. Your company's multi-billion dollar calculator is brought to
|
|
its knees by a simple copy-paste error. You wonder, "how in Sam Hill are these
|
|
people making any money???" (The answer, of course, is that they are a big
|
|
enterprise corporation)
|
|
|
|
You let your boss know about the bad news, you are immediately given any
|
|
resource in the company that you need to get this mission-critical problem
|
|
solved for *any input*. Yesterday. Without breaking the API that the rest of
|
|
the program has hard-coded in.
|
|
|
|
---
|
|
|
|
Let's look at what is common about all these functions. The `add*` family of
|
|
functions seems to all be doing one thing consistently: adding one number to
|
|
another.
|
|
|
|
Let's define a function called `add` that adds any two numbers:
|
|
|
|
```Python
|
|
def add(x, y):
|
|
return x + y
|
|
```
|
|
|
|
This is nice, but it won't work for the task we were given, which is to not
|
|
break the API.
|
|
|
|
Let's go over what a function is in Python. We can define a function as
|
|
something that takes some set of Python values and produces some set of Python
|
|
values:
|
|
|
|
```haskell
|
|
PythonFunction :: [PythonValue] -> [PythonValue]
|
|
```
|
|
|
|
We can read this as "a Python function takes a set of Python values and
|
|
produces a set of Python values". Now we need to define what a Python value
|
|
actually is. To keep things simple, we're only going to define the following
|
|
types of values:
|
|
|
|
- `None` -> no value
|
|
- `Int` -> any whole number (Python calls this `int`)
|
|
- `Text` -> any string value (Python calls this `str`)
|
|
- `Function` -> something that takes and produces values
|
|
|
|
Python [itself has a lot more types that any value can be](https://docs.Python.org/3.4/library/stdtypes.html),
|
|
but for the scope of this blog post, this will do just fine.
|
|
|
|
Now, since a function can return a value and a function is a value, let's see
|
|
what happens if you return a function:
|
|
|
|
```python
|
|
def outer():
|
|
def inner():
|
|
return "Hello!"
|
|
return inner
|
|
```
|
|
|
|
And in the repl:
|
|
|
|
```
|
|
>>> type(outer)
|
|
<type 'function'>
|
|
```
|
|
|
|
So `outer` is a function as we expect. It takes `None` (in Python, a function
|
|
without arguments has `None` for the type of them) and returns a function that
|
|
takes `None` and that function returns `Text` containing `"Hello!"`.
|
|
Let's make sure of this:
|
|
|
|
```
|
|
>>> outer()()
|
|
'Hello!'
|
|
>>> type(outer()())
|
|
<type 'str'>
|
|
```
|
|
|
|
Yay! When nothing is applied to the result of applying nothing to `outer`, it
|
|
returns the `Text` value `"Hello!"`. We can define the type of `outer` as the
|
|
following:
|
|
|
|
```haskell
|
|
outer :: None -> None -> Text
|
|
```
|
|
|
|
Now, let's use this for addition:
|
|
|
|
```python
|
|
# add :: Int -> Int -> Int
|
|
def add(x):
|
|
def inner(y):
|
|
return x + y
|
|
|
|
return inner
|
|
```
|
|
|
|
And in the repl:
|
|
|
|
```
|
|
>>> add(4)(5)
|
|
9
|
|
```
|
|
|
|
A cool feature about this is that now we can dip into something called Partial
|
|
Application. Partial application lets you apply part of the arguments of
|
|
a function and you get another function out of it. Let's trace the type of the
|
|
`inner` function inside the `add` function, as well as the final computation
|
|
for clarity:
|
|
|
|
```python
|
|
# add :: Int -> Int -> Int
|
|
def add(x):
|
|
# inner :: Int -> Int
|
|
def inner(y):
|
|
return x + y # :: Int
|
|
|
|
return inner
|
|
```
|
|
|
|
Starting from the inside, we can see how the core computation here is `x + y`,
|
|
which returns an `Int`. Then we can see that `y` is passed in and in the scope
|
|
also as an `Int`. Then we can also see that `x` is passed in the outermost
|
|
layer as an int, giving it the type `Int -> Int -> Int`. Since `inner` is
|
|
a value, and a Python variable can contain any Python value, let's make
|
|
a function called `increment` using the `add` function:
|
|
|
|
```python
|
|
# increment :: Int -> Int
|
|
increment = add(1)
|
|
```
|
|
|
|
And in the repl:
|
|
|
|
```
|
|
>>> increment(50)
|
|
51
|
|
```
|
|
|
|
`increment` takes the integer given and increases it by 1, it is the same thing
|
|
as defining:
|
|
|
|
```python
|
|
def increment50():
|
|
return 51
|
|
```
|
|
|
|
Or even `51` directly.
|
|
|
|
Now, let's see how we can use this for the `add*` family of function mentioned
|
|
above:
|
|
|
|
```python
|
|
# add :: Int -> Int -> Int
|
|
def add(x):
|
|
def inner(y):
|
|
return x + y
|
|
|
|
return inner
|
|
|
|
# add1 :: Int -> Int
|
|
add1 = add(1)
|
|
|
|
# add2 :: Int -> Int
|
|
add2 = add(2)
|
|
|
|
# add3 :: Int -> Int
|
|
add3 = add(3)
|
|
|
|
# add4 :: Int -> Int
|
|
add4 = add(4)
|
|
```
|
|
|
|
And all we need to do from here is a few simple tests to prove it will work:
|
|
|
|
```python
|
|
if __name__ == "__main__":
|
|
assert add(1)(1) == 2 # 1 + 1
|
|
assert add(1)(2) == add(2)(1) # 1+2 == 2+1
|
|
print("all tests passed")
|
|
```
|
|
|
|
```console
|
|
$ python addn.py
|
|
all tests passed
|
|
```
|
|
|
|
Bam. The `add*` family of functions is now a set of partial applications. It is
|
|
just a set of half-filled out forms.
|
|
|
|
---
|
|
|
|
You easily mechanically rewrite all of the `add*` family of functions to use
|
|
the metaprogramming style you learned on your own. Your patch goes in for
|
|
consideration to the code review team. Meanwhile your teammates are frantically
|
|
going through every function in the 200,000 line file that defines the `add*`
|
|
family of functions. They are estimating months of fixing is needed not to
|
|
mention millions of lines of test code. They are also estimating an additional
|
|
budget of contractors being brought in to speed all this up. Your code has made
|
|
all of this unneeded.
|
|
|
|
Your single commit was one of the biggest in company history. Billboards that
|
|
were red are now beaming a bright green. Your code fixed 5,000 other copy-paste
|
|
errors that have existed in the product for years. You immediately get a raise
|
|
and live happily ever after, a master in your craft.
|
|
|
|
---
|
|
|
|
For fun, let's rewrite the `add` function in Haskell.
|
|
|
|
```haskell
|
|
add :: Int -> Int -> Int
|
|
add x y = x + y
|
|
```
|
|
|
|
And then we can create a partial application with only:
|
|
|
|
```haskell
|
|
add1 :: Int -> Int
|
|
add1 = (add 1)
|
|
```
|
|
|
|
And use it in the repl:
|
|
|
|
```
|
|
Prelude> add1 3
|
|
4
|
|
```
|
|
|
|
Experienced haskellers would probably gawk at this. Because functions are the
|
|
base data type in Haskell, and partial application means that you can make
|
|
functions out of functions, we can define `add` as literally the addition
|
|
operator `(+)`:
|
|
|
|
```haskell
|
|
add :: Int -> Int -> Int
|
|
add = (+)
|
|
```
|
|
|
|
And because operators are just functions, we can further simplify the `add1`
|
|
function by partially applying the addition operation:
|
|
|
|
```haskell
|
|
add1 :: Int -> Int
|
|
add1 = (+1)
|
|
```
|
|
|
|
And that will give us the same thing.
|
|
|
|
```
|
|
Prelude> let add1 = (+1)
|
|
Prelude> add1 3
|
|
4
|
|
```
|
|
|
|
---
|
|
|
|
Now, real world example time. I recently wrote a simple JSON api based off of
|
|
a lot of data that has been marginally useful to some people. This api has
|
|
a series of HTTP endpoints that return data about My Little Pony: Friendship is
|
|
Magic episodes. Its code is [here](https://github.com/Xe/ponyapi) and its
|
|
endpoint is `http://ponyapi.apps.xeserv.us`.
|
|
|
|
One of the challenges when implementing it was how to avoid a massive amount of
|
|
copy-pasted code when doing so. I had started with a bunch of functions like:
|
|
|
|
```python
|
|
# all_episodes :: IO [Episode]
|
|
def all_episodes():
|
|
r = requests.get(API_ENDPOINT + "/all")
|
|
|
|
if r.status_code != 200:
|
|
raise Exception("Not found or server error")
|
|
|
|
return r.json()["episodes"]
|
|
```
|
|
|
|
Which was great and all, but there was so much code duplication involved to
|
|
just get one result for all the endpoints. My first step was to write something
|
|
that just automated the getting of json from an endpoint in the same way
|
|
I automated addition above:
|
|
|
|
```python
|
|
# _base_get :: Text -> None -> IO (Either Episode [Episode])
|
|
def _base_get(endpoint):
|
|
def doer():
|
|
r = requests.get(API_ENDPOINT + endpoint)
|
|
|
|
if r.status_code != 200:
|
|
raise Exception("Not found or server error")
|
|
|
|
try:
|
|
return r.json()["episodes"]
|
|
except:
|
|
return r.json()["episode"]
|
|
|
|
# all_episodes :: IO [Episode]
|
|
all_episodes = _base_get("/all")
|
|
```
|
|
|
|
Where `_base_get` returned the function that satisfied the request.
|
|
|
|
This didn't end up working so well with the endpoints
|
|
[that take parameters](https://github.com/Xe/ponyapi#seasonnumber), so I had to
|
|
account for that in my code:
|
|
|
|
```python
|
|
# _base_get :: Text -> Maybe [Text] -> (Maybe [Text] -> IO (Either Episode [Episode]))
|
|
# _base_get takes a text, a splatted list of texts and returns a function such that
|
|
# the function takes a splatted list of texts and returns either an Episode or
|
|
# a list of Episode as an IO action.
|
|
def _base_get(endpoint, *fragments):
|
|
def doer(*args):
|
|
r = None
|
|
|
|
assert len(fragments) == len(args)
|
|
|
|
if len(fragments) == 0:
|
|
r = requests.get(API_ENDPOINT + endpoint)
|
|
else:
|
|
url = API_ENDPOINT + endpoint
|
|
|
|
for i in range(len(fragments)):
|
|
url = url + "/" + fragments[i] + "/" + str(args[i])
|
|
|
|
r = requests.get(url)
|
|
|
|
if r.status_code != 200:
|
|
raise Exception("Not found or server error")
|
|
|
|
try:
|
|
return r.json()["episodes"]
|
|
except:
|
|
return r.json()["episode"]
|
|
|
|
return doer
|
|
|
|
# all_episodes :: IO [Episode]
|
|
all_episodes = _base_get("/all")
|
|
|
|
# newest :: IO Episode
|
|
newest = _base_get("/newest")
|
|
|
|
# last_aired :: IO Episode
|
|
last_aired = _base_get("/last_aired")
|
|
|
|
# random :: IO Episode
|
|
random = _base_get("/random")
|
|
|
|
# get_season :: Int -> IO [Episode]
|
|
get_season = _base_get("", "season")
|
|
|
|
# get_episode :: Int -> Int -> IO Episode
|
|
get_episode = _base_get("", "season", "episode")
|
|
```
|
|
|
|
And that was it, save the `/search` route, which was acceptable to implement by
|
|
hand:
|
|
|
|
```python
|
|
# search :: Text -> IO [Episode]
|
|
def search(query):
|
|
params = {"q": query}
|
|
r = requests.get(API_ENDPOINT + "/search", params=params)
|
|
|
|
if r.status_code != 200:
|
|
raise Exception("Not found or server error")
|
|
|
|
return r.json()["episodes"]
|
|
```
|
|
|
|
---
|
|
|
|
Months later you have been promoted as high as you can go. You've been teaching
|
|
the other engineers at ACMECorp metaprogramming and even convinced management
|
|
to let the next big project be in Haskell.
|
|
|
|
You are set for life. You have won.
|
|
|
|
---
|
|
|
|
For comments on this article, please feel free to email me, poke me in `#geek`
|
|
on `irc.ponychat.net` (my nick is Xena), or leave thoughts at one of the below
|
|
places this article has been posted.
|
|
|
|
Comments:
|
|
|
|
- [Hacker News](https://news.ycombinator.com/item?id=10127389)
|
|
- [Reddit](https://www.reddit.com/r/programming/comments/3ijyyz/metaprogramming_partial_application_and_currying/)
|