REXGEN Air: converting logs automatically with AWS Lambda

Deploy the Influx Technology conversion function to AWS Lambda so that every RXD log your loggers upload to S3 is decoded and converted automatically.

What the function does

The function is a free example application built on REXLIBRARY, the Influx Technology file conversion library. Its source is on GitHub. It runs in AWS Lambda, the serverless compute service, so there is no server for you to manage. Each time a logger uploads an RXD file to your bucket, S3 triggers the function. The function:

• Decodes the log with the DBC file for each CAN bus
• Writes the outputs you enable: ASAM MDF4, BLF, CSV and Parquet
• Can also write to an InfluxDB time series database, and to Amazon Timestream for existing setups

It works on the RXD logs that REXGEN Air and REXGEN Pro loggers upload to S3.

Before you start, the loggers must already upload to your bucket. See REXGEN Air: uploading logs to Amazon S3. Create the function in the same AWS region as the bucket (and the Timestream database, if you use one).

This guide follows the newer function set: AWS_Lambda.zip, the .NET 8 runtime and one DBC file per bus.

Create a role for the function

The function needs an IAM role that lets it read the logs and DBC files from your bucket and write the converted files back.

1. Create a role in IAM. Select AWS service as the trusted entity and Lambda as the use case. For the steps, see the AWS guide creating a role for an AWS service.
2. Attach the permissions. Add AWSLambdaBasicExecutionRole so the function can write its logs. Add a policy that allows reading and writing objects in your logger bucket only. Add Timestream access only if you enable that output.
3. Name the role and create it. You select it when you create the function.

Do not attach the full-access policies for S3 and Timestream. They work, but they give the function far more access than it needs.

Deploy from the AWS console

1. Create the function. Open Lambda and select Create function.
2. Name the function. Select Author from scratch and enter a function name.
3. Set the runtime and architecture. Select .NET 8 (C#/F#/PowerShell) as the runtime and x86_64 as the architecture.
4. Select the role. Under Permissions, expand Change default execution role. Select Use an existing role and select the role you created. Then select Create function.
5. Upload the code. On the Code tab, select Upload from > .zip file. Select the conversion function zip file and select Save.

   Download the conversion function (ZIP, 13.7 MB)

6. Set the handler. Under Runtime settings, select Edit. Check that the runtime is .NET 8. Set the handler to AWSLambdaFileConvert::AWSLambdaConvert.Function::ConvertFiles and select Save.
7. Give the function room to work. On the Configuration tab, open General configuration and select Edit. Raise the memory, ephemeral storage and timeout from their defaults to suit your logs and DBC files. Large DBC files need more.

Add the S3 trigger

1. Add a trigger. In the function overview, select Add trigger.
2. Set the source. Select S3 as the source, then select the bucket your loggers upload to.
3. Set the suffix. Enter .rxd so the function runs only for new logs, not for the files it writes.
4. Confirm the trigger. Acknowledge the recursive invocation warning and select Add.

Choose the outputs

The function reads its settings from FileConvert.json at the root of the bucket. Copy the example, set enabled to true for each output you want and upload the file to the bucket root.

{
  "InfluxDB": {
    "enabled": false,
    "url": "",
    "token": "",
    "org": "",
    "bucket": ""
  },
  "CSV": {
    "enabled": false,
    "resampling": {
      "enabled": true,
      "rate": 1000
    }
  },
  "AmazonTimestream": {
    "enabled": false,
    "db_name": "test_demo",
    "table_name": "rxddata",
    "downsampling": true,
    "downsampling_tolerance": 0.01
  },
  "Snapshot": {
    "enabled": false,
    "db_name": "dbname",
    "table_name": "snapshot"
  },
  "MDF": {
    "enabled": false,
    "usecompression": false
  },
  "BLF": {
    "enabled": false
  },
  "Parquet": {
    "enabled": true,
    "resampling": {
      "enabled": true,
      "rate": 1000
    }
  },
  "Synch": {
    "enabled": false,
    "main_logger": "",
    "addon_loger1": "",
    "lastfolder": ""
  },
  "RxdSettings": {
    "MaxTimeGap": 0
  }
}

MDF, BLF, CSV, Parquet

File outputs. usecompression compresses MDF4 files. resampling puts CSV and Parquet values on a fixed rate, which you set with rate. Parquet is enabled by default because the Grafana dashboard in the next guide reads it.

InfluxDB

Writes the decoded signals to an InfluxDB database at url, using your token, org and bucket. Keep the token out of shared copies of the file.

AmazonTimestream

Writes decoded signals to an Amazon Timestream table, with optional downsampling. It is kept for existing setups, because new AWS accounts may not be able to create Timestream for LiveAnalytics databases.

Snapshot

Writes the logger's snapshot values to the database table named by db_name and table_name.

Synch and RxdSettings

Advanced settings. Leave them at the values in the example unless an Influx Technology engineer advises otherwise.

Add your DBC files

Put the DBC files in the logger's folder, which the logger creates in the bucket on its first upload. Name each file for the bus it decodes:

• dbc_can0.dbc for CAN 0
• dbc_can1.dbc for CAN 1
• dbc_can2.dbc for CAN 2
• dbc_can3.dbc for CAN 3

Older versions of the function use a single file named ExportDBC.dbc instead. If you deploy an older function, use that name.

New logs are converted as they arrive. Look for the outputs in the logger's date folders, for example a Parquet folder inside each day's folder.

For developers: deploy from Visual Studio or the command line

Use this route to change the function or to deploy it from your own build. The project is the AWS_Lambda demo in the REXLIBRARY repository on GitHub.

From Visual Studio

1. Install Visual Studio 2022. Include the components listed in the Influx Technology configuration file. Download the Visual Studio configuration (VSCONFIG, 5 KB)
2. Open the project. Clone or download the repository and open \Demo\AWS_Lambda\AWSLambdaFileConvert.csproj.
3. Set up AWS SDK credentials. The credentials need permission for S3 and Lambda. For the steps, see the AWS guide providing AWS credentials. Use your own profile, region and role, never values that came with a sample.
4. Start the publish. Right-click the project and select Publish to AWS Lambda.
5. Complete the publish window. Select your AWS credential and the region of your bucket. Add a description if you want one. Select the role you created and leave the other fields at their defaults.
6. Upload the function. Select Upload. Then add the S3 trigger, the conversion settings and the DBC files in the same way as for the console route.

From the command line

Install the Amazon Lambda tools, or update them if they are already installed. Then deploy from the project folder:

dotnet tool install -g Amazon.Lambda.Tools
dotnet tool update -g Amazon.Lambda.Tools
cd "Demo/AWS_Lambda"
dotnet lambda deploy-function

To read RXD files in your own software instead, see REXLIBRARY: convert RXD logs and XML configurations in your own code.

Next: REXGEN Air: Grafana dashboards with Parquet, Glue and Athena.