Buffer formatting

Modules
	Field Formatting Functions
	Set of pre-defined formatting functions.

Data Structures
struct	format_t
	Overall parameters of a format session. More...
struct	formatField_t
	Information about each field to be read or written. More...

Enumerations
enum	formatDirection_t { FORMAT_PACK, FORMAT_UNPACK }
	Used as parameter to the formatting functions to inform if they should handle packing or unpacking. More...

Functions
int	formatPack (format_t *fmt, uint8_t *output, int maxOutput)
	Write to output the packed values as described in fmt. More...
int	formatUnpack (format_t *fmt, uint8_t *input, int inputSize)
	Read from input and unpack the fields to fmt. More...

Detailed Description

Rationale

One of the basic operations necessary to perform during communication and storage is generating a formatted buffer coalesced from many data items, perhaps with necessary conversions, for example ISO8583 message packing. This module is an extension of existing ISO8583 message-packing libraries, with additions so it can be used for other packing formats, and can be highly customized by the user.

Introduction

Formatting (or packing) a buffer is the generation of a buffer from a collection of (many) data elements possibly spread in many data structures, with added conversion and formatting. The inverse process is called unpacking.

The design of this module is such that the user needs to create and maintain a single table with the list of fields, and how to format each, and this same structure can be used both for packing and unpacking. Some pre-defined functions for this are included with the library (see Field Formatting Functions), but the user is free to create new ones.

See the documentation of formatField_t and format_t for information on how to fill those structures. See below for more information about how each field is used during packing and unpacking.

Packing

On a call to formatPack() the following algorithm is followed for each field:

1. If filter != NULL and filter(fmt, field) == 0, ignore this field
2. Set bodySize = 0;
3. If body != NULL, call it to write the field data;
   • Set bodySize to the return value of body;
4. If prefix != NULL, call it;
   • Note that prefix can access bodySize to know the number of bytes actually written by body, and size for the user-provided size information;
5. If suffix != NULL, call it;
   • Note that suffix can access bodySize to know the number of bytes actually written by body, and size for the user-provided size information;
6. Swap body and prefix so prefix is before body on the output buffer.

• The formatting of the size and padding is independent of the formatting of the actual field data;
• bodySize rarely needs to be filled by the user, it is filled on runtime with the value returned by body;
• Fields that require padding to specific sizes can break this into a body that write the actual data and a prefix and/or suffix that deals with the padding;
• On the call to prefix or suffix the size field indicates the number given by the caller when filling the formatField_t structure, while bodySize is the actual number of bytes written by body;
• Padding is usually done by a suffix that writes size - bodySize bytes;
• prefix can be used to write a size field with the actual number of bytes written, as stored in bodySize;
• A separator can be generated using the suffix function.

Unpacking

On a call to formatUnpack() the following algorithm is followed for each field:

1. If filter != NULL and filter(fmt, field) == 0, ignore this field
2. Set bodySize = size;
3. If prefix != NULL, call it;
   • If prefix deals with the actual field length, it must fill bodySize with the number of bytes that need to be read!
4. If body != NULL, call it to read the field data;
   • body should always read bodySize bytes, not size bytes;
5. If suffix != NULL, call it;
   • Note that suffix can access bodySize to know the actual size of body in bytes;

• Remember that prefix should set the bodySize field, and that body must use bodySize instead of size to know how many bytes to read (if prefix is NULL then bodySize = size);
• If suffix is used for padding or separator, then it may only skip the necessary number of bytes;

Examples

Here we list a sample of common field types and a simple ISO8583-like field list with a few bits.

Fixed-size Fields

Fixed-size fields are fields where the number of bytes written by body is always equal to size as given by the user on the field table. This usually means that those fields include no prefix or suffix, but they may require a sufix function to write a fixed separator.

Example: a fixed-size field with 16 bytes.

field.index = fieldIndex;
field.prefix = NULL;
field.body = formatBodyMemcpy;   // copy exactly field.size bytes
field.suffix = NULL;
field.size = 16;                 // size must be provided
field.data = dataPtr;
field.bodySize = 0;

Variable-size Fields

Variable-size fields require a size prefix to indicate the actual size of the body. Usually they are string data (as card-holder name) or numerical data without padding.

Example: a variable-size field of a string.

field.index = fieldIndex;
field.prefix = formatSizeLL;     // size as 2-digit decimal number
field.body = formatBodyStrcpy;   // copy until '\0'
field.suffix = NULL;
field.size = 0;                  // size is not relevant
field.data = stringPtr;
field.bodySize = 0;

Padded Fields

Padded fields are a combination of fixed and variable-sized fields. They have a size prefix but are padded to fit exactly size bytes.

Example: a string field padded with spaces to 20 bytes.

formatField_t field;
field.index = fieldIndex;
field.prefix = formatSizeLL;     // size as 2-digit decimal number
field.body = formatBodyStrcpy;   // copy until '\0'
field.suffix = formatPadSpaces;  // pad with spaces
field.size = 20;                 // desired total size of field
field.data = stringPtr;
field.bodySize = 0;

Filtering

By filling the filter field of the format_t structure, the caller may allow conditional processing of fields. The most common use is during ISO8583 processing, but the system is flexible enough for other uses. During execution the filter function is called for each field, passing a pointer to both the format_t and the specific formatField_t.

A filter implementation may use the fieldMap pointer to store the set of allowed fields. For example, for ISO8583, the fieldMap could be a pointer to the actual ISO8583 bitmap as in the packed buffer.

A common framework is to use a body writer function that both pack/ unpack the bitmap and set the fieldMap field, for example:

int filterISO8583(const format_t *fmt, const formatField_t *field)
{
 return (fmt->fieldMap == NULL)
    || (field->index < 0)
    || bitsGet(fmt->fieldMap, field->index) == 1;
}

Is a function that checks the bit field->index of fmt->fieldMap to choose if a field should or not be part of the packing/unpacking process.

In order for this to work, the user must also define, inside the field list, where the bitmap is stored, and properly associate the value of the fieldMap field. The following function does both:

int formatBodyISO8583(uint8_t *buffer, int limit, formatDirection_t direction, format_t *fmt, formatField_t *field)
{
 int ret;
 if (direction == FORMAT_PACK) {
    memcpy(buffer, field->data, field->size);
    ret = field->size;
  }
 else {
    memcpy(field->data, buffer, field->bodySize);
    ret = field->bodySize;
  }
  fmt->fieldMap = field->data;
 return ret;
}

And can be used as:

bitsBuffer_t iso_bitmap[NBITMAP];
formatField_t field = { -1, NULL, formatBodyISO8583, NULL, sizeof(iso_bitmap), iso_bitmap, 0 };

A simple body function

This is a simple function that could be used as the body of a field. It copies variable-sized zero-terminated strings:

int formatBodyStrcpy(uint8_t *buffer,
 int limit,
 formatDirection_t direction,
 format_t *fmt,
 formatField_t *field)
{
 if (direction == FORMAT_PACK) {
 int n = strlen(field->data);
    memcpy(buffer, field->data, n);
 return n;
  }
 else {
    memcpy(field->data, buffer, field->bodySize);
    ((char *)field->data)[field->bodySize] = '\0';
 return field->bodySize;
  }
}

Note how it return strlen() on packing, and both uses and return bodySize on unpack.

A simple prefix function

Following the example above, this is a prefix function that write the value of bodySize as a two-digit decimal number:

int formatSizeAsc2(uint8_t *buffer,
 int limit,
 formatDirection_t direction,
 format_t *fmt,
 formatField_t *field)
{
 if (direction == FORMAT_PACK) {
 if (limit < 2)
 return BASE_ERR_OVERFLOW;
 if (field->bodySize < 0 || field->bodySize > 99)
 return BASE_ERR_INVALID_PARAMETER;
 convIntToTxtPad(field->bodySize, (char *) buffer, 2, 10);
 return 2;
  }
 else {
 uint64_t value;
 convTxtToInt((const char *) buffer, 2, 10, &value);
    field->bodySize = (int) value;
 return 2;
  }
}

Note how it return the number of bytes read or written on both pack and unpack, and how it uses bodySize on packing and stores to it on unpacking. (This version uses the functions from Conversion routines).

A simple suffix function

Now a function that does padding with spaces: it writes enough spaces to compensate by the different between size and bodySize, if any.

int formatPadSpaces(uint8_t *buffer,
 int limit,
 formatDirection_t direction,
 format_t *fmt,
 formatField_t *field)
{
 if (direction == FORMAT_PACK) {
     memset(buffer, ' ', field->size - field->bodySize);
  }
 return field->size - field->bodySize;
}

Note how, on unpacking, the function ignores the actual buffer contents and just return how many bytes need to be skipped over.

A (small) complete example

Putting together the pieces above we can create a simple formatter table for a buffer with only variable-sized string fields.

First, define the list of fields and their format:

char string1[NSTRING1];
char string2[NSTRING2];
char iso_bitmap[NBITMAP];
formatField_t fields[] = {
 // position of the bitmap on the formatted buffer
  { -1, NULL, formatBodyISO8583, NULL, sizeof(iso_bitmap), iso_bitmap, 0 },
 // no padding, with size prefix
  {  0, formatSizeLL, formatBodyStrcpy, NULL, 0, string1, 0 },
 // padding with spaces
  {  1, formatSizeLL, formatBodyStrcpy, formatPadSpaces, NSTRING2, string2, 0 },
};

Create the format_t instance with the fields defined above:

format_t fmt = {
 // list of fields
  fields,
 // number of fields
 sizeof(fields) / sizeof(fields[0]),
 // filter function
  &formatFilterISO8583,
 // the other fields are initialized to zero
};

Set the bits on iso_bitmap to identify which fields are part of the buffer. (When unpacking, the buffer is read from the input buffer).

// destination buffer
uint8_t output[NOUTPUT];
// clear list of bits
bitsSetRange(iso_bitmap, 0, 8 * NBITMAP, 0);
// enable field indexes 0 and 1
bitsSet(iso_bitmap, 0, 1);
bitsSet(iso_bitmap, 1, 1);
// call formatting
formatPack(&fmt, output, sizeof(output));

Enumeration Type Documentation

enum formatDirection_t

Used as parameter to the formatting functions to inform if they should handle packing or unpacking.

Enumerator
FORMAT_PACK	Write to buffer.
FORMAT_UNPACK	Read from buffer.

Function Documentation

int formatPack	(	format_t *	fmt,
		uint8_t *	output,
		int	maxOutput
	)

Write to output the packed values as described in fmt.

Parameters

fmt	format_t with packing parameters.
output	Where to store the packed message.
maxOutput	Size of output, in bytes.

Returns

Number of bytes writen on success.

Negative on error.

int formatUnpack	(	format_t *	fmt,
		uint8_t *	input,
		int	inputSize
	)

Read from input and unpack the fields to fmt.

Parameters

fmt	format_t with packing parameters.
input	Where to read the packed message.
inputSize	Size of input, in bytes.

Returns

Number of bytes actually read from input, on success.

Negative on error.