Data-Acquisition Preprocessor

Data-Acquisition Preprocessor on t6 is a process done in 8 distinct phases prior to data analysis itself.

Tagged on #preprocessor, #feature,

Phase 1-3 : Filter and securize

• Signature validation aims to make sure the payload originated from the expected source and has not been tampered.
• Encryption aims to make sure only t6 api is able to read the content of the payload.

The two first steps (sigature & encryption) can be combined together. On the third step, any payload that does not fit either signature validation or encryption read will be rejected and not performed in the Data Integration process.

Phase 4 : Processors Modules

Each Flow can contain the preprocessor attribute (optional) as an array of preprocessor(s). Or, by adding a Datapoints, the payload can have this preprocessor attribute as well. Payload preprocessor is overwritting the Flow when both are having the attribute.

During phase 4, preprocessor can do several modifications and controls on measures: transformation, convertion, sanitization, validation and Automatic identification and data capture (AIDC).

Transformation processors

Incoming values posted to Flows can be modified by 1 or multiple preprocessor(s). So before it comes to the Rule engine and before any storage on TimeseriesDb. The available transformers are the following:

On th following table, the input is Consequatur quis veniam natus ut qui..

In the case of mode=“aes-256-cbc”, the payload must contains an extra attribute: object_id with the value of the Object which knows the secret.

E.g:

"preprocessor": [
    {
        "name": "transform",
        "mode": "snakeCase"
    }
]

An additional transform preprocessor is flexible enough to susbstitute/replace string based on regexp. E.g:

"value": "John Smith",
"preprocessor": [
    {
        "name": "transform",
        "mode": "replace",
        "pattern": "(\\w+)\\s(\\w+)",
        "replacer": "$2, $1"
    }
]

As a consequence t6 will transform the passed value from John Smith to Smith, John.

Convertion processors

The convert preprocessor is a very simple unit converter for the main following units:

• time
• distance
• mass
• volume
• storage
• things
• temperature  not implemented

E.g:

"preprocessor": [
        {
            "name": "convert",
            "type": "distance",
            "from": "km",
            "to": "m"
        }
    ]

Sanitization processors

The sanitize preprocessor is aiming to make sure the value is using the expected Datatype.

Note: This sanitization is forced from t6 according to the Flow, but it can also be added manually if you’d manually need to sanitize prior any other processor.

E.g:

"preprocessor": [
        {
            "name": "sanitize",
            "datatype": "float"
        }
    ]

An additional attribute can be added when Adding a [Datapoint](/features/Datapoints) to *t6* and containing the uuiv-v4 of the corresponding [Datatype](/features/data-types/). In this case the (automatically added) Sanitization will use the specified [Datatype](/features/data-types/).

E.g: 
```json
"data_type": "e7dbdc23-5fa8-4083-b3ec-bb99c08a2a35",

Validation processors

Validation on preprocessor aims to validate the value sent to t6 and/or reject from any storage in case the value does not pass validation test. Validation allows Decision Rule to follow up on the value.

The validation are the following :

E.g:

"preprocessor": [
    {
        "name": "validate",
        "test": "isEmail"
    }
]

Automatic Identification and Data Capture (AIDC)

AIDC preprocessor expect to deals with images preprocessing to identify objects, faces and facial expressions.

AIDC modes are the following :

E.g:

"preprocessor": [
        {
            "name": "aidc",
            "mode": "faceExpressionRecognition",
        }
    ]