layline.io Blog

ASN.1 Format Configuration

ASN.1 is still a popular data format. Learn how easy it is to configure any ASN.1 format in layline.io.

May 11, 2022
Reading time: 5 min.

Intro to ASN.1

In our last article “/Damn that Data Format Hell/” we talked about how layline.io deals with the multitude of complex structural data formats by using a grammar language to describe almost any such format, and enabling you to instantly work with the data.

We also discussed, that this doesn’t work for ALL formats you may be working with, but maybe 80% of all structured formats.

There are data exchange formats out there in the wild, however, which adhere to their own standard and come with their own grammar language. One of which is Abstract Syntax Notation One, aka ASN.1. The grammar notation is described in various standardization documents. To describe them is beyond the scope of this piece, but you can research them with the link provided below.

layline.io Showcase

In this article we will showcase how our users can define ASN.1 based formats using layline.io.

layline.io is an opinionated event data processor. It wraps the power of reactive stream management in a framework which makes setup, deployment and monitoring of large scale event data processing requirements easy and straightforward. layline.io provides everything you would expect from a product-base solution including upgradability, carrier-grade resilience and scalability, among other features.

TD.57 - Transferred Account Procedure (TAP)

One popular area where ASN.1 protocol format plays an important role is in telecoms roaming and the ensuing TAP exchange process. TAP (Transferred Accounts Procedure) handles the financial settlement process between mobile networks for their roaming subscribers.

In a nutshell: If you use your cell phone in a visiting network (e.g. other country or simply other network), then the visited network needs to invoice your home network for the service provided. The information about who used their phone in a visited network, how long and who they called, etc. is all exchanged in TAP and RAP files. These file types are both based on ASN.1. Their grammar is well-defined in the TD.57 specification which you can find here.

Simplified TD.57 TAP procedure
Simplified TD.57 TAP procedure

Our example: TAP3.11 file format configuration

In our example, we will configure the quite complex format of TAP3.11 which travels on line (1) in the image above (remember, it could be any other ASN.1-based format).

TAP3.11 Grammar

This is what the beginning of the official TAP3.11 grammar looks like:

-- TD57_3.11.2
--
-- The following ASN.1 specification defines the abstract syntax for
--        Data Record Format Version 03
--                           Release 11
...

TAP-0311  DEFINITIONS IMPLICIT TAGS  ::=

BEGIN

DataInterChange ::= CHOICE
{
    transferBatch TransferBatch,
    notification  Notification,
...
}
...
TransferBatch ::= [APPLICATION 1] SEQUENCE
{
    batchControlInfo       BatchControlInfo            OPTIONAL, -- *m.m.
    accountingInfo         AccountingInfo              OPTIONAL,
    networkInfo            NetworkInfo                 OPTIONAL, -- *m.m.
    messageDescriptionInfo MessageDescriptionInfoList  OPTIONAL,
    callEventDetails       CallEventDetailList         OPTIONAL, -- *m.m.
    auditControlInfo       AuditControlInfo            OPTIONAL, -- *m.m.
...
}

Create ASN.1 Asset

Given the above, let’s set up TAP3.11 using layline.io. (We will not explain how to fully set up a Project in the article. If you are interested in this then please check the respective documentation in the resource links at the end of this blog).

In the Project Asset View (1) pick Formats (2) and then ASN.1 Format (3).

Create ASN.1 Asset
Create ASN.1 Asset

The Asset editor has two tabs “Configuration” and “Modules”. On the “Configuration” tab (1) we name the Asset “TAP3.11” (2):

Naming the new Asset
Naming the new Asset

Define Message Types

TAP3.11 defines a number of different “root” message types. We are only going to identify three of them to work with in this example:

Available Message Types
Available Message Types

In layline.io we map them to “Header”, “Detail”, and “Trailer” respectively it looks like this (1):

Defining Message Types in layline.io
Defining Message Types in layline.io

We are later going to debug output the data we read. Because some data is BCD-encoded and for readability’s sake we use a built-in converter to decode them and make them readable:

BCD En-/Decoding
BCD En-/Decoding

Adding the TAP3.11 grammar

We are now ready to add the actual TAP3.11 grammar. Switch to the Modules tab (2) of the ASN.1 Asset we have added (1).

An ASN.1 grammar can be made-up of multiple modules. In our case it is just one. We have added it and named it “TAP.311-Module” (3/4).

Adding ASN.1 grammar module
Adding ASN.1 grammar module

You can find the original ASN.1 grammar for TAP3.11 in the written specification:

GSMA's TD.57 TAP3.11 ASN.1 Grammar Notation
GSMA's TD.57 TAP3.11 ASN.1 Grammar Notation

Let’s copy and paste it to our ASN.1 Asset (1) (it’s about 1,422 lines):

Pasting ASN.1 Grammar into layline.io
Pasting ASN.1 Grammar into layline.io

Interactive Testing of the ASN.1 Grammar

layline.io provides the ability to instantly check whether format setup matches the actual data. You can upload a sample file (1) directly into the Configuration Center (2) to perform a cross-check.

Uploading ASN.1 Sample File
Uploading ASN.1 Sample File

Let’s do this (2 above):

Viewing ASN.1 Sample File
Viewing ASN.1 Sample File

layline.io will instantly

  1. Generate a parser for the ASN.1 grammar
  2. Parse the sample file
  3. Feed the result back (3 & 4)

The file content will be displayed in a token tree (3) as well as in a hex-dump style display (4). You can mark individual tokens (3) to see the corresponding file content highlighted (4).

Switch to the “Sample Messages” tab (2) where we can get a more readable view on the file content:

Decoded ASN.1 Sample File Viewing
Decoded ASN.1 Sample File Viewing

Interactive testing means, that you now could go back to the grammar, change it, and then instantly see how this affects how this affects the interpretation of your sample data.

Sample Project

We put together a complete sample Project together with test-data which is ready to run in layline.io. The example simply reads and writes a TAP file and outputs its contents to the console in a JSON-format:

Sample Project TAP3 Workflow
Sample Project TAP3 Workflow
Sample Project Debug Output
Sample Project Debug Output

Any other complex operation on the data would work as well, of course and is likely to be what would be happening in a real Project context. But for the purpose of our sample, the above will suffice.

You can find the sample Project in the resource links at the end of this article.

Summary

Using layline.io you can not only define complex, distributed, scalable event-processing scenarios, but pretty much work with any data format. In this example we demonstrated how to deal with ASN.1 structured data.

Most other solutions will support specific formats only, as their parsers have been hard-coded to match a specific use-case. In layline.io there are literally millions of use-cases. That’s why formats are never hard-coded, but configurable.

There are many popular ASN.1 protocols around the world of which we would like to list some below. And there are - of course - countless custom ASN.1 protocols as well.

Protocol

  • Ericsson AIR / AXE10 / CCN / CCR / MTAS / R3 / R4 / R7 / R8 / R9 / R10 / R11 / R12 / R13 / SDP / Tele2 TIP
  • Huawei IMS
  • 4G UMTS / 5G IMS / 3GPP
  • MetaSolv GPRS
  • TD.35 NRTRDE - Near Real Time Roaming Data Exchange
  • TD.57 / TAP2 / TAP3 Transferred Account Procedure
  • Interledger Protocol
  • NTCIP 1103 - Transport Management Protocols
  • X.500 Directory Services
  • Lightweight Directory Access Protocol (LDAP)
  • PKCS Cryptography Standards
  • X.400 Message Handling
  • T.120 Multimedia conferencing
  • Simple Network Management Protocol (SNMP)
  • Common Management Information Protocol (CMIP)
  • Signalling System No. 7 (SS7)
  • ITU H-Series Multimedia Protocols
  • BioAPI Interworking Protocol (BIP)
  • Common Biometric Exchange Formats Framework (CBEFF)
  • Authentication Contexts for Biometrics (ACBio)
  • Computer-supported telecommunications applications (CSTA)
  • Dedicated short-range communications (DSRC)
  • IEEE 802.11p (IEEE WAVE)
  • Intelligent Transport Systems (ETSI ITS)
  • Global System for Mobile Communications (GSM)
  • General Packet Radio Service (GPRS) / Enhanced Data rates for GSM Evolution (EDGE)
  • Universal Mobile Telecommunications System (UMTS)
  • Long-Term Evolution (LTE)
  • Common Alerting Protocol (CAP)
  • Controller–pilot data link communications (CPDLC)
  • Space Link Extension Services (SLE)
  • Manufacturing Message Specification (MMS)
  • File Transfer, Access and Management (FTAM)
  • Remote Operations Service Element protocol (ROSE)
  • Association Control Service Element (ACSE)
  • Building Automation and Control Networks Protocol (BACnet)

More

There is a lot more to tell here. For example what the hell is layline.io anyway? In the links below you can learn about layline.io and also dive deeper into the documentation of how this specific grammars work in layline.io.

You can also grab your free copy of layline.io from our website and work with a sample Project which you can also download.

Thanks for reading!

Resources

#Project files
1Github: Simple ASN.1 Project (input test files in the directory _test_files of the Project)
#Documentation
1Documentation:Getting Started
2Documentation:Generic Format Asset?
3ASN.1 on Wikipedia

Previous

Damn that Data Format Hell

Next

Sniffing Real-Time Data