t6 Features > Data-Acquisition Preprocessor

Data-Acquisition Preprocessor

Data-Acquisition Preprocessor on t6 is a process done in 8 distinct phases prior to data analysis itself.

t6 Data integration process

Phase 1-3 : Filter and securize

The two first steps (sigature & encryption) can be combined together. On the third step, any payload that does not fit either signature validation or encryption read will be rejected and not performed in the Data Integration process.

Phase 4 : Processors Modules

Each Flow can contain the preprocessor attribute (optional) as an array of preprocessor(s). Or, by adding a Datapoints, the payload can have this preprocessor attribute as well. Payload preprocessor is overwritting the Flow when both are having the attribute.

During phase 4, preprocessor can do several modifications and controls on measures: transformationconvertionsanitizationvalidation and Automatic identification and data capture (AIDC).

Transformation processors

Incoming values posted to Flows can be modified by 1 or multiple preprocessor(s). So before it comes to the Rule engine and before any storage on TimeseriesDb. The available transformers are the following:

Mode Example input Example output
camelCase Consequatur quis veniam natus ut qui. consequaturQuisVeniamNatusUtQui
capitalCase Consequatur quis veniam natus ut qui. Consequatur Quis Veniam Natus Ut Qui
constantCase Consequatur quis veniam natus ut qui. CONSEQUATUR_QUIS_VENIAM_NATUS_UT_QUI
dotCase Consequatur quis veniam natus ut qui. consequatur.quis.veniam.natus.ut.qui
headerCase Consequatur quis veniam natus ut qui. Consequatur-Quis-Veniam-Natus-Ut-Qui
noCase Consequatur quis veniam natus ut qui. consequatur quis veniam natus ut qui
paramCase Consequatur quis veniam natus ut qui. consequatur-quis-veniam-natus-ut-qui
pascalCase Consequatur quis veniam natus ut qui. ConsequaturQuisVeniamNatusUtQui
pathCase Consequatur quis veniam natus ut qui. consequatur/quis/veniam/natus/ut/qui
sentenceCase Consequatur quis veniam natus ut qui. Consequatur quis veniam natus ut qui
snakeCase Consequatur quis veniam natus ut qui. consequatur_quis_veniam_natus_ut_qui
upperCase Consequatur quis veniam natus ut qui. CONSEQUATUR QUIS VENIAM NATUS UT QUI.
aes-256-cbc Consequatur quis veniam natus ut qui. e.g.: f5596c8aa3bfaf03882554760518e4b7:f30aaa6f54e175d2a4579eb228[... ...]9469810ba0a495d1643. The string before : refers to vector iv

In the case of mode=“aes-256-cbc”, the payload must contains an extra attribute: object_id with the value of the Object which knows the secret.

E.g:

"preprocessor": [
    {
        "name": "transform",
        "mode": "snakeCase"
    }
]

Convertion processors

The convert preprocessor is a very simple unit converter for the main following units:

E.g:

"preprocessor": [
        {
            "name": "convert",
            "type": "distance",
            "from": "km",
            "to": "m"
        }
    ]

Sanitization processors

The sanitize preprocessor is aiming to make sure the value is using the expected Datatype.

Note: This sanitization is forced from t6 according to the Flow, but it can also be added manually if you’d manually need to sanitize prior any other processor.

E.g:

"preprocessor": [
        {
            "name": "sanitize",
            "datatype": "float"
        }
    ]

An additional attribute can be added when Adding a [Datapoint](/features/Datapoints) to *t6* and containing the uuiv-v4 of the corresponding [Datatype](/features/data-types/). In this case the (automatically added) Sanitization will use the specified [Datatype](/features/data-types/).

E.g: 
```json
"data_type": "e7dbdc23-5fa8-4083-b3ec-bb99c08a2a35",

Validation processors

Validation on preprocessor aims to validate the value sent to t6 and/or reject from any storage in case the value does not pass validation test. Validation allows Decision Rule to follow up on the value.

The validation are the following :

Test Example input Result
isEmail rejectedEmail@______domain.com ❌ due to invalid domain name
isEmail AcCePtEdEmail@domain.com
isAscii Rejected char 😀 ❌ due to invalid character
isBase32 aBCDE23=
isBase32 89gq6t9k68==
isBase58 aBCDE23=
isBase58 a4E9kYnK==
isBase64 aBCDE23==
isBase64 QmFzZTY0==
isBIC ABC-FR23GHI
isBIC ABCFRPP
isBoolean Truezz
isBoolean True

E.g:

"preprocessor": [
    {
        "name": "validate",
        "test": "isEmail"
    }
]

Automatic Identification and Data Capture (AIDC)

AIDC preprocessor expect to deals with images preprocessing to identify objects, faces and facial expressions.

AIDC modes are the following :

Mode Purpose
faceExpressionRecognition Identify the best match expression (neutral, happy, sad, angry, fearful, disgusted, surprised) from the 1st face detected in the image
genderRecognition Identify the gender (male, female) from the 1st face detected in the image
ageRecognition Identify the age from the 1st face detected in the image

E.g:

"preprocessor": [
        {
            "name": "aidc",
            "mode": "faceExpressionRecognition",
        }
    ]