Tracing

Tracing Exporters

The tracing module supports several exporters, custom tracing spans and integration with instrumentation libraries (like dd-trace). Set tracing: true in broker’s options to enable tracing.

Enable tracing

const broker = new ServiceBroker({
    tracing: true
});

Console

This is a debugging exporter which prints full local trace to the console.

Console exporter can’t follow remote calls, only locals.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            {
                type: "Console",
                options: {
                    logger: null,
                    colors: true,
                    width: 100,
                    gaugeWidth: 40
                }
            }
        ]
    }
});

Datadog

Datadog exporter sends tracing data to Datadog server via dd-trace. It is able to merge tracing spans of instrumented Node.js modules and Moleculer modules.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            {
                type: "Datadog",
                options: {
                    agentUrl: process.env.DD_AGENT_URL || "http://localhost:8126",
                    env: process.env.DD_ENVIRONMENT || null,
                    samplingPriority: "AUTO_KEEP",
                    defaultTags: null,
                    tracerOptions: null,
                }
            }
        ]
    }
});

To use this exporter, install the dd-trace module with npm install dd-trace --save command.

Event

Event exporter sends Moleculer events ($tracing.spans) with tracing data.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            {
                type: "Event",
                options: {
                    eventName: "$tracing.spans",

                    sendStartSpan: false,
                    sendFinishSpan: true,

                    broadcast: false,

                    groups: null,

                    /** @type {Number} Batch send time interval. */
                    interval: 5,

                    spanConverter: null,

                    /** @type {Object?} Default span tags */
                    defaultTags: null

                }
            }
        ]
    }
});

Event (legacy)

Legacy event exporter sends Moleculer legacy metric events (metrics.trace.span.start & metrics.trace.span.finish) at every request. These events are also used to generate metrics in legacy (<= v0.13) metrics solutions.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            "EventLegacy"
        ]
    }
});

Legacy Request Started Payload

The broker emits an metrics.trace.span.start event when a new request is started.
The payload looks like the following:

{
    // Context ID
    id: '4563b09f-04cf-4891-bc2c-f26f80c3f91e',
    // Request ID
    requestID: '6858979d-3298-4a7b-813a-ffb417da822b',
    // Level of call
    level: 1,
    // Start time
    startTime: 1493903164726,
    // Is it a remote call
    remoteCall: false,
    // Called action
    action: {
        name: 'users.get'
    },
    // Called service
    service: {
        name: "users"
    },
    // Params
    params: {
        id: 5
    },
    // Meta
    meta: {},
    // Node ID
    nodeID: "node-1",
    // Caller nodeID if it's requested from a remote node
    callerNodeID: "node-2",
    // Parent context ID if it is a sub-call
    parentID: null
}

Legacy Request Finished Payload

The broker emits an metrics.trace.span.finish event when the call/request is finished.
The payload looks like the following:

{
    // Context ID
    id: '4563b09f-04cf-4891-bc2c-f26f80c3f91e',
    // Request ID
    requestID: '6858979d-3298-4a7b-813a-ffb417da822b',
    // Level of call
    level: 1,
    // Start time
    startTime: 1493903164726,
    // End time
    endTime: 1493903164731.3684,
    // Duration of request
    duration: 5.368304,
    // Is it a remote call
    remoteCall: false,
    // Is it resolved from cache
    fromCache: false,
    // Called action
    action: {
        name: 'users.get'
    },
    // Called service
    service: {
        name: "users"
    },
    // Params
    params: {
        id: 5
    },
    // Meta
    meta: {},   
    // Node ID
    nodeID: "node-1",
    // Caller nodeID if it's a remote call
    callerNodeID: "node-2",
    // Parent context ID if it is a sub-call
    parentID: null,
    // Error if the call returned with error
    error: {
        name: "ValidationError",
        message: "Invalid incoming parameters"
    }
}

Jaeger

Jaeger exporter sends tracing spans information to a Jaeger server.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            {
                type: "Jaeger",
                options: {
                    /** @type {String?} HTTP Reporter endpoint. If set, HTTP Reporter will be used. */
			        endpoint: null,
                    /** @type {String} UDP Sender host option. */
                    host: "127.0.0.1",
                    /** @type {Number?} UDP Sender port option. */
                    port: 6832,

                    /** @type {Object?} Sampler configuration. */
                    sampler: {
                        /** @type {String?} Sampler type */
                        type: "Const",

                        /** @type: {Object?} Sampler specific options. */
                        options: {}
                    },

                    /** @type {Object?} Additional options for `Jaeger.Tracer` */
                    tracerOptions: {},

                    /** @type {Object?} Default span tags */
                    defaultTags: null
                }
            }
        ]
    }
});

To use this exporter, install the jaeger-client module with npm install jaeger-client --save command.

Zipkin

Zipkin exporter sends tracing spans information to a Zipkin server.

const broker = new ServiceBroker({
    tracing: {
        enabled: true,
        exporter: [
            {
                type: "Zipkin",
                options: {
                    /** @type {String} Base URL for Zipkin server. */
                    baseURL: "http://localhost:9411",

                    /** @type {Number} Batch send time interval. */
                    interval: 5,

                    /** @type {Object} Additional payload options. */
                    payloadOptions: {

                        /** @type {Boolean} Set `debug` property in v2 payload. */
                        debug: false,

                        /** @type {Boolean} Set `shared` property in v2 payload. */
                        shared: false
                    },

                    /** @type {Object?} Default span tags */
                    defaultTags: null
                }
            }
        ]
    }
});

Customizing

Multiple Spans per Action

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        async find(ctx) {
            const span1 = ctx.startSpan("get data from DB", {
                tags: {
                    ...ctx.params
                }
            }); 
            const data = await this.getDataFromDB(ctx.params);
            ctx.finishSpan(span1);

            const span2 = ctx.startSpan("populating");
            const res = await this.populate(data);
            ctx.finishSpan(span2);

            return res;
        }
    }
};

Adding Context Values

You can customize what context params or meta values are added to span tags.

Default

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                // Add all params without meta
                tags: {
                    params: true,
                    meta: false,
            },
            async handler(ctx) {
                // ...
            }
        }
    }
});

Custom params example

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                // Add `ctx.params.id` and `ctx.meta.loggedIn.username` values
                // to tracing span tags.
                tags: {
                    params: ["id"],
                    meta: ["loggedIn.username"],
                    response: ["id", "title"] // add data to tags from the action response.
            },
            async handler(ctx) {
                // ...
            }
        }
    }
});

Example with custom function
Please note, the tags function will be called two times in case of success execution. First with ctx, and second times with ctx & response as the response of action call.

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        get: {
            tracing: {
                tags(ctx, response) {
                    return {
                        params: ctx.params,
                        meta: ctx.meta,
                        custom: {
                            a: 5
                        },
                        response
                    };
                }
            },
            async handler(ctx) {
                // ...
            }
        }
    }
});

Example of Event tracing

// posts.service.js
module.exports = {
    name: "posts",
    events: {
        "user.created": {
            tracing: {
                // Add all params without meta
                tags: {
                    params: true,
                    meta: false,
            },
            async handler(ctx) {
                // ...
            }
        }
    }
});