BitMagic-C++
Public Types | Public Member Functions | Protected Types | Protected Member Functions | Static Protected Member Functions | Protected Attributes

#include <bmsparsevec_serial.h>

Collaboration diagram for bm::sparse_vector_serializer< SV >:
Collaboration graph
[legend]

Public Types

typedef SV::bvector_type bvector_type
 
typedef const bvector_typebvector_type_const_ptr
 
typedef bvector_typebvector_type_ptr
 
typedef SV::value_type value_type
 
typedef SV::size_type size_type
 
typedef bvector_type::allocator_type alloc_type
 
typedef alloc_type::allocator_pool_type allocator_pool_type
 
typedef bm::serializer< bvector_type >::bv_ref_vector_type bv_ref_vector_type
 
typedef bm::serializer< bvector_type >::xor_sim_model_type xor_sim_model_type
 

Public Member Functions

 sparse_vector_serializer ()
  More...
 
Compression settings <br>
void set_bookmarks (bool enable, unsigned bm_interval=256) BMNOEXCEPT
 Add skip-markers for faster range deserialization. More...
 
void enable_xor_compression () BMNOEXCEPT
 Enable XOR compression on vector serialization. More...
 
void disable_xor_compression () BMNOEXCEPT
 Disable XOR compression on serialization. More...
 
void set_xor_ref (bool is_enabled) BMNOEXCEPT
 Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector. More...
 
void set_xor_ref (const bv_ref_vector_type *bv_ref_ptr) BMNOEXCEPT
 Enable external XOR serialization via external reference vectors (data frame ref. More...
 
void compute_sim_model (xor_sim_model_type &sim_model, const bv_ref_vector_type &ref_vect, const bm::xor_sim_params &params)
 Calculate XOR similarity model for ref_vector refernece vector must be associated before. More...
 
void set_sim_model (const xor_sim_model_type *sim_model) BMNOEXCEPT
 Attach serizalizer to a pre-computed similarity model. More...
 
bool is_xor_ref () const BMNOEXCEPT
 Returns the XOR reference compression status (enabled/disabled) More...
 
Serialization <br>
void serialize (const SV &sv, sparse_vector_serial_layout< SV > &sv_layout)
 Serialize sparse vector into a memory buffer(s) structure. More...
 
bm::serializer< bvector_type > & get_bv_serializer () BMNOEXCEPT
 Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings. More...
 

Protected Types

typedef SV::remap_matrix_type remap_matrix_type
 
typedef bm::heap_vector< unsigned, alloc_type, true > u32_vector_type
 
typedef bm::serializer< bvector_typeserializer_type
 
typedef serializer_type::buffer buffer_type
 

Protected Member Functions

void build_xor_ref_vector (const SV &sv)
 
void encode_remap_matrix (bm::encoder &enc, const SV &sv)
 serialize the remap matrix used for SV encoding More...
 

Static Protected Member Functions

static void build_plane_digest (bvector_type &digest_bv, const SV &sv)
 

Protected Attributes

bm::serializer< bvector_typebvs_
  More...
 
bvector_type plane_digest_bv_
 bv.digest of bit-planes More...
 
buffer_type plane_digest_buf_
 serialization buf More...
 
u32_vector_type plane_off_vect_
 
u32_vector_type remap_rlen_vect_
 
bool is_xor_ref_
  More...
 
bv_ref_vector_type bv_ref_
 
xor_sim_model_type sim_model_
 
const bv_ref_vector_typebv_ref_ptr_
 
const xor_sim_model_typesim_model_ptr_
 

Detailed Description

template<typename SV>
class bm::sparse_vector_serializer< SV >
\brief Serialize sparse vector into a memory buffer(s) structure

Serialization format:

| HEADER | BIT-VECTORS ... | REMAP_MATRIX

Header structure:

BYTE+BYTE: Magic-signature 'BM' or 'BC' (c-compressed) BYTE : Byte order ( 0 - Big Endian, 1 - Little Endian) { BYTE : Number of Bit-vector planes (total) (non-zero when < 255 planes) | BYTE: zero - flag of large plane matrix INT64: Nnmber of bit-vector planes } INT64: Vector size INT64: Offset of plane 0 from the header start (value 0 means plane is empty) INT64: Offset of plane 1 from ... INT32: reserved

Bit-vectors:

Based on current bit-vector serialization

Remap Matrix: SubHeader | Matrix BLOB

sub-header: BYTE: 'R' (remapping) or 'N' (no remapping) N - means no other info is saved on the stream INT64: remap matrix size

Examples
rscsample02.cpp, rscsample05.cpp, strsvsample05.cpp, svsample02.cpp, and svsample08.cpp.

Definition at line 175 of file bmsparsevec_serial.h.

Member Function Documentation

◆ compute_sim_model()

template<typename SV >
void bm::sparse_vector_serializer< SV >::compute_sim_model ( xor_sim_model_type sim_model,
const bv_ref_vector_type ref_vect,
const bm::xor_sim_params params 
)

Calculate XOR similarity model for ref_vector refernece vector must be associated before.

See also
set_ref_vectors, set_sim_model
Examples
rscsample05.cpp.

Definition at line 801 of file bmsparsevec_serial.h.

Referenced by serialize_df2().

◆ disable_xor_compression()

template<typename SV >
void bm::sparse_vector_serializer< SV >::disable_xor_compression ( )
inline

Disable XOR compression on serialization.

Examples
svsample02.cpp.

Definition at line 221 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::set_xor_ref().

Referenced by SDemo2().

◆ enable_xor_compression()

template<typename SV >
void bm::sparse_vector_serializer< SV >::enable_xor_compression ( )
inline

Enable XOR compression on vector serialization.

See also
set_xor_ref
disable_xor_compression
Examples
svsample02.cpp.

Definition at line 215 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::set_xor_ref().

Referenced by SDemo2().

◆ encode_remap_matrix()

template<typename SV >
void bm::sparse_vector_serializer< SV >::encode_remap_matrix ( bm::encoder enc,
const SV &  sv 
)
protected

◆ get_bv_serializer()

template<typename SV >
bm::serializer<bvector_type>& bm::sparse_vector_serializer< SV >::get_bv_serializer ( )
inline

Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings.

See also
bm::serializer::set_compression_level

Definition at line 281 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::bvs_.

◆ is_xor_ref()

template<typename SV >
bool bm::sparse_vector_serializer< SV >::is_xor_ref ( ) const
inline

Returns the XOR reference compression status (enabled/disabled)

Examples
svsample02.cpp.

Definition at line 260 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::is_xor_ref_.

Referenced by SDemo2().

◆ serialize()

template<typename SV >
void bm::sparse_vector_serializer< SV >::serialize ( const SV &  sv,
sparse_vector_serial_layout< SV > &  sv_layout 
)

◆ set_bookmarks()

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_bookmarks ( bool  enable,
unsigned  bm_interval = 256 
)
inline

Add skip-markers for faster range deserialization.

Parameters
enable- TRUE searilization will add bookmark codes
bm_interval- bookmark interval in (number of blocks) (suggested between 4 and 512) smaller interval means more bookmarks added to the skip list thus more increasing the BLOB size
Examples
rscsample02.cpp, strsvsample05.cpp, and svsample08.cpp.

Definition at line 207 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::bvs_, and bm::serializer< BV >::set_bookmarks().

Referenced by main().

◆ set_sim_model()

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_sim_model ( const xor_sim_model_type sim_model)

Attach serizalizer to a pre-computed similarity model.

Parameters
sim_model- pointer to external computed model
Examples
rscsample05.cpp.

Definition at line 812 of file bmsparsevec_serial.h.

Referenced by serialize_df2().

◆ set_xor_ref() [1/2]

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_xor_ref ( bool  is_enabled)

Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector.

Default: disabled Reference bit-vectors from the sparse vector itself

Examples
rscsample05.cpp.

Definition at line 792 of file bmsparsevec_serial.h.

Referenced by bm::sparse_vector_serializer< SV >::disable_xor_compression(), bm::sparse_vector_serializer< SV >::enable_xor_compression(), serialize_df0(), and serialize_df2().

◆ set_xor_ref() [2/2]

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_xor_ref ( const bv_ref_vector_type bv_ref_ptr)

Enable external XOR serialization via external reference vectors (data frame ref.

vector). This method is useful when we serialize a group of related sparse vectors which benefits from the XOR referencial compression

Parameters
bv_ref_ptr- external reference vector if NULL - resets the use of reference vector

Definition at line 781 of file bmsparsevec_serial.h.

Field Documentation

◆ bvs_

template<typename SV >
bm::serializer<bvector_type> bm::sparse_vector_serializer< SV >::bvs_
protected

◆ is_xor_ref_

template<typename SV >
bool bm::sparse_vector_serializer< SV >::is_xor_ref_
protected

◆ plane_digest_buf_

template<typename SV >
buffer_type bm::sparse_vector_serializer< SV >::plane_digest_buf_
protected

serialization buf

Definition at line 309 of file bmsparsevec_serial.h.

◆ plane_digest_bv_

template<typename SV >
bvector_type bm::sparse_vector_serializer< SV >::plane_digest_bv_
protected

bv.digest of bit-planes

Definition at line 308 of file bmsparsevec_serial.h.


The documentation for this class was generated from the following file: