Multimodal Data Representation Model for Complex Description of Observation Objects

Abstract

The paper is devoted to solving the problem of the complex presentation of multimodal data about an object of observation, the characteristics of which are measured and investigated taking into account the time stamps and the relationship between the data of various modalities. This problem is relevant for many engineering tasks. In particular, the comprehensive presentation of data on a physical object is important in the technology of digital twins, which has recently become more widely used. Based on the fact that a digital twin is determined by both a behavioral model and a visual model, the paper presents a data model based on the concept of a multimodal data element. This model makes possible a comprehensive and consistent presentation of data about the object of observation.

The mathematical model of the complex presentation of multimodal data is based on the concepts of an aggregate and a multi-image defined in an algebraic system of aggregates. The article analyzes the relationship between the complex presentation and processing of multimodal data, which are determined in time, using the apparatus of the algebraic system of aggregates and the consideration of aggregated data as functions of several variables. Thus, the analysis of the data of the muxel model can be performed using logical operations, ordering operations, and relations defined in the algebraic system of aggregates as well as operations and approaches defined in other mathematical concepts.

Since the muxel model presents large amounts of data, it is advisable to optimize the way how data is presented. This can be achieved by data compression, modification of the data structure, or a combination of the first and second approaches. In the first approach, it is advisable to apply muxel model data compression based on the RLE method. In the second approach, it is proposed to apply data structures similar to the Sparse Voxel Octree, which is used to reduce data volume of models in voxel graphics. The practical implementation of the processing of multimodal data presented using the muxel model can be performed by means of ASAMPL programming language.