Use Python partial objects to write cleaner higher order functions

The problem

So you want to create a function that will do the following:

1. Call an API.
2. If the call succeeds, call another function passing the result as argument.

A naive solution

You try the following:

def process_data(data):
  # save data to db or something here.
  pass

def fetch_data(callback):
  result = http_client.get('someurl')
  if result.ok:
    callback(result)

fetch_data(process_data)

fetch_data will run and if the request succeeds, it will call process_data, passing the result as argument.

Now, a new requirement came up where you need process_data to know whether it should overwrite existing data or not. So you change the signature like so:

def process_data(data, force=False):
  if force:
    # force processing
    pass
  else:
    # Don't force
    pass

How are you going to pass this to fetch_data now? force defaults to False which is fine but what about when you want it to be True?

Will you change the caller’s signature as follows?

def fetch_data(callback, force_processing=False):
  result = http_client.get('someurl')
  if result.ok:
    callback(result, force_processing)

fetch_data(process_data, force_processing)

Cool, problem solved.

fetch_data can take any callback function, so you receive a new requirement where fetched data should only be logged instead of processed.

You try this out:

def log_data(data, destination):
  if destination == 'console':
    print(data)
  else:
    # Log data somewhere else
    pass

Now you need to modify fetch_data again to account for the destination parameter:

def fetch_data(callback, force_processing=False, destination='console'):
  ...

Every time you add callbacks, your fetch_data caller’s signature will need to be updated as well. Then you have to go through all calling sites to update the signature. This is error-prone and makes refactoring hard.

A not so naive solution

Okay, what if you simply use *args* and **kwargs instead of fixed parameters.

Like so?

def fetch_data(callback, *callback_args, **callback_kwargs):
  callback(*callback_args, **callback_kwargs)

fetch_data(process_data, True)
fetch_data(log_data, destination='syslog')

Ah, this looks much better.

Now, what happens if fetch_data itself needs an argument now?

def fetch_data(url, callback, *callback_args, **callback_kwargs):
  result = http_client.get(url)
  if result.ok:
    callback(*callback_args, **callback_kwargs)

fetch_data('myurl', process_data, True)

This solution works but notice how when you call fetch_data, you don’t really know which arguments are for the callback function and which ones are for the caller.

As your requirements grow, this will become even more confusing.

Enter partial objects

Fortunalety, the Python standard library provides a concept called partial objects.

Instead of passing the function itself as a callback, you turn it into a partial object first and pass that instead.

Let’s reimplement everything:

def process_data(data, force=False):
  pass

def log_data(data, destination):
  pass

def fetch_data(url, callback):
  result = http_client.get(url)
  if result.ok:
    callback()

Notice how we’re now calling callback directly without any arguments in fetch_data. But how can we use this now?

Check this out:

from functools import partial

fetch_data('someapi', partial(process_data, force=true))
fetch_data('anotherapi', partial(log_data, destination='console'))

By passing a partial object instead of the function itself, fetch_data doesn’t need to know anything about the callback’s signature. It can simply call it because the partial object has already been prepared to be called with the proper arguments.

Your code is now much cleaner and easier to refactor.