# Health notifications


Upsun Fixed monitors disk space and inode usage. Learn how health notifications are handled on Grid and Dedicated Gen 2 environments.

## Notification types

Notifications include the steps to take to resolve the issue.

### Low-disk warning

Upsun Fixed monitors disk space usage on all applications and services in your cluster.

* When available disk space drops below 20% or 4 GB, whichever is smaller, a warning notification is generated.
* When available disk space drops below 10% or 2 GB, whichever is smaller, a critical notification is generated.
* When available disk space returns above 20% or 4 GB, whichever is smaller, an all-clear notification is generated.

### Low-inode warning

Inodes are filesystem metadata entries that store information about files and directories.
Each file, directory, or symlink consumes one inode, regardless of its size. Running out of inodes prevents
the creation of new files even when disk space is available.

Upsun Fixed monitors inode usage on all applications and services in your cluster.

* When available inodes drop below 20% of the total available, a warning notification is generated.
* When available inodes drop below 10% of the total available, a critical notification is generated.
* When available inodes return above 20% of the total available, an all-clear notification is generated.

Notifications are generated every 5 minutes, so there may be a brief delay between when the threshold is crossed and when the notification is triggered.

## Health notifications on Grid {#health-notifications-on-grid}

By default, Upsun Fixed emails low-disk and low-inode notifications to [project admins](https://fixed.docs.upsun.com/administration/users.md#project-roles).

As a project admin, you can also:   
- Configure the sending of health notifications through other channels (for example, via Slack or PagerDuty) as described in this topic.
- Use the CLI to [add](https://fixed.docs.upsun.com/administration/cli/reference.md#integrationadd) and [manage](https://fixed.docs.upsun.com/administration/cli/reference.md#integrationupdate) health notifications. 

### Email notifications

A notification can trigger an email to be sent from an address of your choosing to one or more addresses of your choosing.

You can view an email notification by running `platform integration:get`.

```bash
platform integration:get
+--------------+---------------+
| Property     | Value         |
+--------------+---------------+
| id           | abcdefghijklm |
| type         | health.email  |
| role         |               |
| from_address |               |
| recipients   | - '#admins'   |
+--------------+---------------+
```

To edit the `recipients` that receive the default email notification, use the `integration:update` command.

```bash
platform integration:update abcdefghijklm --recipients you@example.com
```

The `recipients` field may be any valid email address, or one of the following special values.

* `#admins` maps to all project admins and up.
* `#viewers` maps to everyone with access to the project.

To add a new email notification, register a `health.email` integration as follows:

```bash
platform integration:add --type health.email --recipients them@example.com --recipients others@example.com
```

You must specify one or more `recipients`, each as its own switch.

The default `from-address` points to the "Upsun Fixed Bot".

You can also configure a custom `--from-address`. The `--from-address` is whatever address you want the email to appear to be from. It is completely fine to use the same email address for both `from-address` and `recipients`. Note that depending on the configuration of the recipient mail server (including SPF and DKIM DNS entries) when using a custom `from-address`, the email can be marked as spam or lost.

### Slack notifications

A notification can trigger a message to be posted to a Slack channel via a [Slack app](https://api.slack.com/apps).

#### 1. Optional: Create a Slack app

**Note**: 

If you are already have a Slack app, you can jump to [enabling notifications](#2-enable-notifications).

1. Open the [Slack API website](https://api.slack.com/) and go to **Your apps**.
2. Click **Create an App**.
3. Choose if you want to build your app from scratch, or via [an app manifest](https://api.slack.com/concepts/manifests).
4. Give your app a name.
5. Select a workspace to install your app in.
   **Note**: 

If you are not an admin of the selected workspace, request approval to install your app there.

6. Click **Create App**.

#### 2. Enable notifications

1. Open the [Slack API website](https://api.slack.com/) and go to **Your apps**.
2. Go to **Features** > **OAuth & Permissions** in the sidebar.
3. Scroll down to the **Scopes** area and select **User Token Scopes**.
4. Click **Add an OAuth Scope** to add the `chat:write` scope.

    ![Slack app scopes](https://fixed.docs.upsun.com/images/slack/slack-app-scopes.png "0.30")

5. Go to **Settings** > **Install app** in the sidebar.
   **Note**: 

If you are not an admin of the workspace, you need to provide a link to install the app into the workspace and onto a channel.

Once the app is approved and installed, a **User OAuth Token** is provided in your app settings.
Copy the token, and use it in the next step.

6. Using the **User OAuth Token**, run the following command:

    ```bash
    platform integration:add --type health.slack --token <USER_OAUTH_TOKEN> --channel <CHANNEL_NAME> --project <PROJECT_ID>
    ```

    For example, if you want your Slack app to post messages in the `project-notifications` channel, write the channel name in the command as you would reference it within Slack:

    ```bash
    platform integration:add --type health.slack ... --channel '#project-notifications' ...
    ```
6. When the integration is successfully configured, Upsun Fixed then sends an initial message to the channel.

**Bot users v. Slack apps**: 

Previously, Upsun Fixed allowed for the configuration of health notifications sent to Slack via bot users and their associated API tokens.
As of June 2024, Slack has deprecated bot users, and integrations must be configured using Slack apps as described above.

If you already have defined an integration using a bot user API token, it will continue to work properly, though you should consider upgrading your processes to the above settings to avoid any future retirement.

### PagerDuty notifications

A notification can trigger a message to be sent via PagerDuty, if you are using that service.
First, create a new PagerDuty "[integration](https://support.pagerduty.com/docs/services-and-integrations)" that uses the Events API v2.
Copy the "Integration Key" as known as the "routing key" for the integration.

Now register a `health.pagerduty` integration as follows:

```bash
platform integration:add --type health.pagerduty --routing-key YOUR_ROUTING_KEY
```

Any notification will now trigger an alert in PagerDuty.

### Validate the integration

You can then verify that your integration is functioning properly using the [`integration:validate`](https://fixed.docs.upsun.com/administration/cli/reference.md#integrationvalidate) command

```bash
platform integration:validate
```

## Health notifications on Dedicated Gen 2  

For Dedicated Generation 2 projects, the response to health notifications depends on the environment type.

| Environment | Response |
|-------------|----------|
| Production (any cloud) | Upsun Support and Operations teams are alerted and a support ticket is automatically opened on your behalf. |
| Staging  | No alerts are sent and no action is taken by Upsun. You can monitor your disk usage via the [Console](https://fixed.docs.upsun.com/administration/web.md) or [CLI](https://fixed.docs.upsun.com/administration/cli.md). |
| Development/preview | DG2 development/preview environments run on the Grid - notifications work as described in the [Health notifications on Grid](#health-notifications-on-grid) section of this topic. |

## Related topics
- [Dedicated Gen 2 Overview](dedicated-environments/dedicated-gen-2/overview.md)
- [Dedicated Gen 2 Development](https://fixed.docs.upsun.com/dedicated-environments/dedicated-gen-2/development.md)