How to set up external monitoring and email alerts using Node.js

Deephaven’s persistent queries should be, as the name suggests, persistent. But that doesn’t mean they are immune from errors. Maybe an update didn’t go as planned, or your server is running out of resources to allocate daily queries, or your query ran out of RAM. This could lead to problematic failures. Thankfully, Deephaven puts troubleshooting methods within reach. One such option is to monitor Deephaven externally using OpenAPI and Node.js.

Monitoring can be done in 4 simple steps:

  1. Connect to a Deephaven instance (learn about this here).
  2. Configure the notification service.
  3. Listen for failures.
  4. Send failure notifications to email, Slack, etc.

This guide shows an example of how to use Deephaven’s OpenAPI with Node.js to easily monitor persistent query status and send alert emails when they encounter errors or failures.

Connecting to Deephaven

We will be using the base project from How to set up a Deephaven OpenAPI application in Node.js as a starting point. This tutorial is assuming you already have some persistent queries you'd like to monitor. Note that the code in app.js uses example values for the server location and credentials that would need to be changed to match your authentication details.

Below is some example code based on the starter code for connecting to Deephaven. After connecting, it gets the notification config (the configuration details) and then creates our failure monitor.

const nodemailer = require('nodemailer');
const openapiInclude = require('./openapiIncludeAsync');

const host = 'server.name';
const credentials = { username: 'username', token: 'secret', type: 'password' };
const apiFileName = 'irisapi.nocache.js';

start();
async function start() {
  try {
    await openapiInclude(`https://${host}/irisapi/${apiFileName}`, apiFileName);
    const wsUrl = `wss://${host}/socket`;

    let client = new iris.Client(wsUrl);
    client.addEventListener('connect', async () => {
      await client.login(credentials);
      console.log('Authenticated');

      let transporter = await getNotificationConfig();
      new PQAlertMonitor(client, transporter);
    });
  } catch (err) {
    console.error(err);
  }
}

Configuring the Notification Service

After we connect to Deephaven, we need to configure our notification service. For email notifications, the Nodemailer package is a good choice. To use Nodemailer, we will have to create a transport object with our email server options. This transporter can then be used to send messages.

To simplify testing our email (and prevent spamming our inbox), we will use a test account from Ethereal, which is conveniently included with Nodemailer. Ethereal lets you preview emails in your browser without sending the email. If you want to send real emails, then the transporter needs to reflect your actual email server configuration.

The code for creating the Nodemailer transporter is below.

async function getNotificationConfig() {
  // Gets a random test account from ethereal
  const testAccount = await nodemailer.createTestAccount();

  const transporter = nodemailer.createTransport({
    host: 'smtp.ethereal.email',
    port: 587,
    secure: false,
    auth: {
      user: testAccount.user,
      pass: testAccount.pass,
    },
  });

  return transporter;
}

Listening for Failures

Now we need to monitor our persistent queries for failures. Each persistent query can exist in one of several states including:

  • AcquiringWorker
  • Initializing
  • Running
  • Failed
  • Error
  • Disconnected
  • Stopped

The current state is one part of the query’s configuration. Conveniently, OpenAPI provides a method to listen for config updates which are often triggered by state changes. We can then check the state on each config update to see if the query has moved to one of the failing states (“Disconnected”, “Error”, or “Failed”) and send a notification.

For this example, we will create a class called “PQAlertMonitor” to monitor for failures and send notifications. In the constructor for this class, we will add our event listener for persistent query config updates so the monitoring starts as soon as we create an instance of the class. The constructor code is below.

class PQAlertMonitor {
 constructor(client, transporter) {
   this.transporter = transporter;
   this.client = client;
   this.client.addEventListener(iris.Client.EVENT_CONFIG_UPDATED, e => this.onConfigUpdate(e));
   console.log('Monitoring...');
 }
...
}

As mentioned above, there are three bad states we'd want to be notified about: “Disconnected”, “Error”, and “Failed”. We will also need to gather some extra info about the query that failed from the config, which will help in troubleshooting. A few useful config properties are name - the human-readable name for the query (e.g., “My Query”) - and the serial - a unique identifier that is more useful to quickly pinpoint the exact query in logs.

The code to check if the query has a bad status is below:

class PQAlertMonitor {
...
 /**
  * Checks if the update is a failure and sends a message if it is
  */
 async onConfigUpdate(event) {
   console.log('Got config updated event', event.detail.name, event.detail.status);

   switch (event.detail.status) {
     case 'Disconnected':
     case 'Error':
     case 'Failed': {
       this.sendMessage(event);
       break;
     }
   }
 }
...
}

Sending Failure Messages

The final part of our monitoring system is to actually send a message. We will just send a simple message containing the name of our query, serial, and query host. These pieces of information should be enough to pinpoint exactly which query you’re looking for even if there are hundreds of running persistent queries. (You might want to check out our documentation on Deephaven's internal tables, such as the QueryPerformanceLog, to learn more about finding the details on your query and how to address its bad state.)

The code to send an email is shown below:

class PQAlertMonitor {
...
 /**
  * Sends a message for a given event
  */
 async sendMessage(event) {
   let info = await this.transporter.sendMail({
     from: 'Test Failure <test@failure.com>',
     to: 'failure@notifications.com',
     subject: `[Deephaven] PQ ${event.detail.name} ${event.detail.status}`,
     text: `Name: ${event.detail.name}\nSerial: ${event.detail.serial}\nHost: ${host}`
   });

   console.log(`Preview URL: ${nodemailer.getTestMessageUrl(info)}`);
 }
}

These messages could easily be sent with any other service (Slack, SMS, etc.) by switching up the messenger object and corresponding functions. We’re using Ethereal to easily preview our email formatting and, if needed, make changes without spamming our inbox. Nodemailer lets you retrieve a URL to the message on Ethereal. An example message preview is shown below:

img

Once you’re satisfied with the formatting of your email, you can change the Nodemailer config to use an actual email server instead of Ethereal.

In this guide, we covered how to dynamically grab the latest API file from the server, how to connect and authenticate with Deephaven OpenAPI in NodeJS, and how to send a simple email with details about a Deephaven persistent query that ran into a problem. (And of course, you can also set up alerts about anything, such as table updates.) You never want your system to fail, but when it does, it’s nice to receive a prompt notification so it can be fixed.