Compactor Compressor objects

Compactor / Compressor objects

These two objects represent the compression of a time series (eg to minimize the volumes of data sent over an LPWAN network).

Compression goals

1 - Replacement of a set of points coded on 16 bits (eg integer) or 32 bits (eg real) into a reduced set of points and coded on a reduced number of bits.

Example: 16 real points (64 bytes) compressed into 12 bytes

2 - Use of easily implementable algorithms on microcontrollers.

Example: use of Sigfox (limitation to 140 messages per day and 12 bytes per message). -> with a measurement every 15 seconds, you can send the 32 values measured every 8 minutes on 12 bytes

Compression principles

Compression is carried out by sending parameters allowing a sequence of several values to be reconstructed (sequence of 32 values in the example above).

The parameters come from several estimations by polynomial regression and are then coded on a defined number of bits (see curve compression).

The compression error (standard deviation between the initial values and the reconstructed values) is also integrated into the compression.

Implementation

Two types of algorithms are implemented (class: Compactor and class: Compressor). They are based on the Serie object and have the following functions:

• functions independent of a dataset:
  • check(): check the input constructor arguments
  • taillePayload(): number of bits of compressed data
  • precisionCodage(): precision of the coding used (maximal error)
  • tauxCompression(): ratio between the number of bits after compression and the number of bits before compression (2 bytes per value)
• functions linked to the dataset:
  • calcul(): generation of the main outputs
  • simul(): simulated values after compression / decompression (usable after calcul)
  • ecartTypeSimul(): standard deviation of simulated values / original values (usable after calcul)
  • param(): getters for parameters (usable after calcul) (Compactor only)
  • compress(): compressed values (usable after calcul)
  • compressEct(): compressed value of standard deviation (usable after calcul)
  • compressYp(): compressed value whithout standard deviation (usable after calcul)
• functions related to decompressing a dataset:
  • decompressY0(): values reconstituted by decompression
  • decompressYp(): parameter values from the compression (Compactor only)
  • decompressEcartType(): standard deviation of simulated values / original values

Principle of simple compression (class: Compactor)

Step 1: Standardization

Scaling of [- 0.5 0.5 ] of the values of the sequence from the minimum, maximum thresholds.

Step 2: Regression

Calculation of the p parameters of the polynomial representing the sequence.

An additional parameter is calculated to represent the performance of the compression (including the deviations linked to the coding): the standard deviation between the initial points and the estimated points.

Step 3: Coding of parameters

The parameters (points + standard deviation) are coded on the number of bits defined.

The result is a bit array (which can then be converted to variables of given length).

Principle of advanced compression (class: Compressor)

A second method combining two regression levels is implemented. It allows in particular to get rid of the minimum / maximum limits which penalize the coding:

1 - Realization of a first regression on the supplied sequence.

2 - Realization of a second series of regressions on the sub-sequences made up of the differences between the initial values and those resulting from the first regression. The points of the second regression are coded in the envelope [- 2 * type deviation, 2 * type deviation ] which makes it possible to reduce the number of coding bits necessary.

Example of a sequence of 32 points: We carry out a first regression with one point (average). For the 32 deviations from the average, perform 4 regressions (polynomials on 3 points) on the 4 sub-sequences of 8 points. On a therefore represented our 32 initial points by 1 + 4 * 3 points. The coding of the 4 * 3 points can be done on a low number of bits, for example 3 or 4 bits while the coding of the average point or the initial standard deviation must be done according to the minimum-maximum range defined.

This advanced compression is based on the simple compression class (Compactor).

Principle of decompression implementation

Decompression consists in calculating the values from the parameters coded with the opposite steps to those of compression:

Step 1: Decoding the parameters

Reconstruction of actual values from the coded value.

Step 2: Decompression

Reconstitution of the estimated values from the parameters of the polynomial.

Step 3: Denormalization

Scaling of estimates based on defined minimum / maximum thresholds

Step 4: Standard deviation

Reconstruction of the compression indicator (standard deviation).

Algorithms used

see curve compression

Use

See the examples given on the two types of regressions.