RPC HOWTO

MAAS contains an RPC mechanism such that every process in the region is connected to every process in the cluster (strictly, every pserv process). It’s based on AMP, specifically Twisted’s implementation.

Where do I start?

Start in the provisioningserver.rpc package. The first two files to look at are cluster.py and region.py. This contain the declarations of what commands are available on clusters and regions respectively.

A new command could be declared like so:

from twisted.protocols import amp

class EatCheez(amp.Command):
    arguments = [
        (b"name", amp.Unicode()),
        (b"origin", amp.Unicode()),
    ]
    response = [
        (b"rating", amp.Integer()),
    ]

It’s also possible to map exceptions across the wire using an errors attribute; see the docs or code for more information.

Note that byte-strings are used for parameter names. Twisted gets quite fussy about this, so remember to do it.

Implementing commands

To implement a new command on the cluster, see the class provisioningserver.rpc.clusterserver.Cluster. A method decorated with @cluster.EatCheez.responder is the implementation of the EatCheez command. There’s no trick to this, they’re just plain old functions. However:

• They only receive named parameters, so the arguments must match the names used in the command’s arguments declaration.
• They must return a dict that matches the command’s response declararation.
• If the response declaration is empty they must still return an empty dict.

To implement a new command on the region, see the class maasserver.rpc.regionserver.Region. It works the same.

Making remote calls from the region to the cluster

There’s a convenient API in maasserver.rpc:

• getClientFor() returns a client for calling remote functions against the cluster identified by a specified UUID.
• getAllClients() will return clients for all connections to cluster processes.

The clients returned are designed to be used in either the reactor thread or in another thread; when called from the latter, a crochet.EventualResult will be returned.

Making remote calls from the cluster to the region

You need to get a handle to the rpc service that will have been started by twistd.

Probably the best way to do this is implement the behaviour you want as a new service, start it up via same mechanism as the rpc service (see provisioningserver.plugin, and pass over a reference.

Then call getClient(), and you will get a client for calling into a region process. You’re given a random client.

Making multiple calls at the same time from outside the reactor

A utility function – gather() – helps here. An example:

from functools import partial

from maasserver.rpc import getAllClients
from maasserver.utils import async
from twisted.python.failure import Failure

# Wrap those calls you want to make into no-argument callables, but
# don't call them yet.
calls = [
    partial(client, EatCheez)
    for client in getAllClients()
]

# Use gather() to issue all the calls simultaneously and process the
# results as they come in. Note that responses can be failures too.
for response in async.gather(calls, timeout=10):
    if isinstance(response, Failure):
        pass  # Do something sensible with this.
    else:
        celebrate_a_cheesy_victory(response)

Responses can be processed as soon as they come in. Any responses not received within timeout seconds will be discarded.

Miscellaneous advice

• Don’t hang onto client objects for long periods of time. It’s okay for a sequence of operations, but don’t keep one around as a global, for example; get a new one each time.
• It’s a distributed system, and errors are going to be normal, so be prepared.

API

Controlling the event-loop in region controllers

Event-loop support for the MAAS Region Controller.

This helps start up a background event loop (using Twisted, via crochet) to handle communications with Cluster Controllers, and any other tasks that are not tied to an HTTP reqoest.

maasserver.eventloop.loop

The single instance of RegionEventLoop that’s all a process needs.

maasserver.eventloop.services

The MultiService which forms the root of this process’s service tree.

This is a convenient reference to loop.services.

maasserver.eventloop.start

Start all the services in services.

This is a convenient reference to loop.start.

maasserver.eventloop.stop

Stop all the services in services.

This is a convenient reference to loop.stop.

RPC declarations for region controllers

RPC declarations for the region.

These are commands that a region controller ought to respond to.

class provisioningserver.rpc.region.ReportBootImages(**kw)

Bases: twisted.protocols.amp.Command

Report boot images available on the invoking cluster controller.

class provisioningserver.rpc.region.Identify(**kw)

Bases: twisted.protocols.amp.Command

Request the identity of the remote side, e.g. its UUID.

RPC implementation for region controllers

class maasserver.rpc.regionservice.Region(boxReceiver=None, locator=None)

Bases: twisted.protocols.amp.AMP

The RPC protocol supported by a region controller.

This can be used on the client or server end of a connection; once a connection is established, AMP is symmetric.

identify()

Implementation of Identify.

report_boot_images(uuid, images)

Implementation of ReportBootImages.

get_tls_parameters()

Implementation of StartTLS.

RPC declarations for cluster controllers

RPC declarations for clusters.

These are commands that a cluster controller ought to respond to.

class provisioningserver.rpc.cluster.ListBootImages(**kw)

Bases: twisted.protocols.amp.Command

List the boot images available on this cluster controller.

class provisioningserver.rpc.cluster.DescribePowerTypes(**kw)

Bases: twisted.protocols.amp.Command

Get a JSON Schema describing this cluster’s power types.

class provisioningserver.rpc.cluster.Identify(**kw)

Bases: twisted.protocols.amp.Command

Request the identity of the remote side, e.g. its UUID.

RPC implementation for cluster controllers

class provisioningserver.rpc.clusterservice.Cluster(boxReceiver=None, locator=None)

Bases: twisted.protocols.amp.AMP, object

The RPC protocol supported by a cluster controller.

This can be used on the client or server end of a connection; once a connection is established, AMP is symmetric.

identify()

Implementation of Identify.

list_boot_images()

Implementation of ListBootImages.

describe_power_types()

Implementation of DescribePowerTypes.

get_tls_parameters()

Implementation of StartTLS.

Helpers

maasserver.rpc.getAllClients()

Get all recorded clients ready to make RPCs to clusters.

maasserver.rpc.getClientFor(uuid)

Get a client with which to make RPCs to the specified cluster.

Raises:NoConnectionsAvailable when there are no open connections to the specified cluster controller.

maasserver.utils.async.gather(calls, timeout=10.0)

Issue calls into the reactor, passing results back to another thread.

Parameters:

• calls – An iterable of no-argument callables to be called in the reactor thread. Each will be called via maybeDeferred().
• timeout – The number of seconds before further results are ignored. Outstanding results will be cancelled.

Returns:

A UseOnceIterator of results. A result might be a failure, i.e. an instance of twisted.python.failure.Failure, or a valid result; it’s up to the caller to check.

ClusterClientService.getClient()

Returns a common.Client connected to a region.

The client is chosen at random.

Raises:NoConnectionsAvailable when there are no open connections to a region controller.