Use Python partial objects to write cleaner higher order functions

I wanted to do the following:

1. Call an API.
2. If the call succeeds, call another function passing the result as argument.

It looks simple, so I solved it as follows:

def process_data(data):
  # save data to db or something here.
  pass

def fetch_data(callback):
  result = http_client.get('someurl')
  if result.ok:
    callback(result)

fetch_data(process_data)

fetch_data will run and if the request succeeds, it will call process_data, passing the result as argument.

Now, a new requirement came up where I need process_data to know whether it should overwrite existing data or not. So I changed the signature like so:

def process_data(data, force=False):
  if force:
    # force processing
    pass
  else:
    # Don't force
    pass

Now I need to figure out how to pass this to fetch_data when force is True.

I can refactor fetch_data to accept an additional parameter and pass it along to processs_data.

def fetch_data(callback, force_processing=False):
  result = http_client.get('someurl')
  if result.ok:
    callback(result, force_processing)

fetch_data(process_data, force_processing)

Looks like I solved the problem, but now a new requirement came up again. Instead of processing the data right away, I need to be able to only log it to a stream instead.

Here’s the log function:

def log_data(data, destination):
  if destination == 'console':
    print(data)
  else:
    # Log data somewhere else
    pass

I can’t simply pass this function as a callback to fetch_data because of the destination parameter.

The only solution is to refactor fetch_data again:

def fetch_data(callback, force_processing=False, destination='console'):
  ...

It seems like every time I add callbacks, fetch_data’s signature needs to be updated. Then, I have to go through all calling sites to update how I’m calling it. This is error-prone and makes refactoring hard.

Arbritary parameters

I want fetch_data to be more flexible, so I can rely on *args* and **kwargs**.

def fetch_data(callback, *callback_args, **callback_kwargs):
  callback(*callback_args, **callback_kwargs)

fetch_data(process_data, True)
fetch_data(log_data, destination='syslog')

Looks cool.

Now a requirement came up where fetch_data needs to accept a url.

def fetch_data(url, callback, *callback_args, **callback_kwargs):
  result = http_client.get(url)
  if result.ok:
    callback(*callback_args, **callback_kwargs)

fetch_data('myurl', process_data, True)

This is confusing me because fetch_data accepts a number of arguments that it will use itself and also other arguments that will be used by the callback function.

When more requirements come up, it will be difficult to refactor this function because I won’t immediately know where each argument is needed.

Partial objects

Python has a package called functools that contains a bunch of goodies that allows you to use functional programming techniques in your code.

Functional programming is great for solving problems related to functions — what I’m dealing with now.

I decided to try using partial objects

Instead of passing the calllback directly, I will first turn it into a partial object and then pass the result.

First, I will remove all parameters needed by the callback from the fetch_data function.

def process_data(data, force=False):
  pass

def log_data(data, destination):
  pass

def fetch_data(url, callback):
  result = http_client.get(url)
  if result.ok:
    callback()

Then I can pass partials to fetch_data.

from functools import partial

fetch_data('someapi', partial(process_data, force=true))
fetch_data('anotherapi', partial(log_data, destination='console'))

Wow, so amazing! fetch_data doesn’t need to know anything about the callback’s signature. This is now easier to make sense of, which is a huge win for maintainability.