BitMagic-C++
Public Types | Public Member Functions | Protected Types | Protected Member Functions | Static Protected Member Functions | Protected Attributes

Serialize sparse vector into a memory buffer(s) structure. More...

#include <bmsparsevec_serial.h>

Collaboration diagram for bm::sparse_vector_serializer< SV >:
Collaboration graph
[legend]

Public Types

typedef SV::bvector_type bvector_type
 
typedef const bvector_typebvector_type_const_ptr
 
typedef bvector_typebvector_type_ptr
 
typedef SV::value_type value_type
 
typedef SV::size_type size_type
 
typedef bvector_type::allocator_type::allocator_pool_type allocator_pool_type
 
typedef bm::serializer< bvector_type >::bv_ref_vector_type bv_ref_vector_type
 
typedef bvector_type::allocator_type alloc_type
 

Public Member Functions

 sparse_vector_serializer ()
 
void serialize (const SV &sv, sparse_vector_serial_layout< SV > &sv_layout)
 Serialize sparse vector into a memory buffer(s) structure. More...
 
bm::serializer< bvector_type > & get_bv_serializer () BMNOEXCEPT
 Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings. More...
 
Compression settings
void set_bookmarks (bool enable, unsigned bm_interval=256)
 Add skip-markers for faster range deserialization. More...
 
void enable_xor_compression () BMNOEXCEPT
 Enable XOR compression on vector serialization. More...
 
void disable_xor_compression () BMNOEXCEPT
 Disable XOR compression on serialization. More...
 
void set_xor_ref (bool is_enabled) BMNOEXCEPT
 Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector. More...
 
void set_xor_ref (const bv_ref_vector_type *bv_ref_ptr) BMNOEXCEPT
 Enable external XOR serialization via external reference vectors (data frame ref. More...
 
bool is_xor_ref () const BMNOEXCEPT
 Returns the XOR reference compression status (enabled/disabled) More...
 

Protected Types

typedef SV::remap_matrix_type remap_matrix_type
 
typedef bm::heap_vector< unsigned, alloc_type, true > u32_vector_type
 
typedef bm::serializer< bvector_typeserializer_type
 
typedef serializer_type::buffer buffer_type
 

Protected Member Functions

void build_xor_ref_vector (const SV &sv)
 
void encode_remap_matrix (bm::encoder &enc, const SV &sv)
 serialize the remap matrix used for SV encoding More...
 

Static Protected Member Functions

static void build_plane_digest (bvector_type &digest_bv, const SV &sv)
 

Protected Attributes

bm::serializer< bvector_typebvs_
 
bvector_type plane_digest_bv_
 bv.digest of bit-planes More...
 
buffer_type plane_digest_buf_
 serialization buf More...
 
u32_vector_type plane_off_vect_
 
u32_vector_type remap_rlen_vect_
 
bool is_xor_ref_
 
bv_ref_vector_type bv_ref_
 
const bv_ref_vector_typebv_ref_ptr_
 

Detailed Description

template<typename SV>
class bm::sparse_vector_serializer< SV >

Serialize sparse vector into a memory buffer(s) structure.

Serialization format:

| HEADER | BIT-VECTORS ... | REMAP_MATRIX

Header structure:

BYTE+BYTE: Magic-signature 'BM' or 'BC' (c-compressed) BYTE : Byte order ( 0 - Big Endian, 1 - Little Endian) { BYTE : Number of Bit-vector plains (total) (non-zero when < 255 plains) | BYTE: zero - flag of large plain matrix INT64: Nnmber of bit-vector plains } INT64: Vector size INT64: Offset of plain 0 from the header start (value 0 means plain is empty) INT64: Offset of plain 1 from ... INT32: reserved

Bit-vectors:

Based on current bit-vector serialization

Remap Matrix: SubHeader | Matrix BLOB

sub-header: BYTE: 'R' (remapping) or 'N' (no remapping) N - means no other info is saved on the stream INT64: remap matrix size

Examples:
rscsample02.cpp, rscsample05.cpp, strsvsample05.cpp, svsample02.cpp, and svsample08.cpp.

Definition at line 160 of file bmsparsevec_serial.h.

Member Typedef Documentation

◆ alloc_type

Definition at line 172 of file bmsparsevec_serial.h.

◆ allocator_pool_type

Definition at line 169 of file bmsparsevec_serial.h.

◆ buffer_type

template<typename SV>
typedef serializer_type::buffer bm::sparse_vector_serializer< SV >::buffer_type
protected

Definition at line 264 of file bmsparsevec_serial.h.

◆ bv_ref_vector_type

Definition at line 171 of file bmsparsevec_serial.h.

◆ bvector_type

template<typename SV>
typedef SV::bvector_type bm::sparse_vector_serializer< SV >::bvector_type

Definition at line 163 of file bmsparsevec_serial.h.

◆ bvector_type_const_ptr

template<typename SV>
typedef const bvector_type* bm::sparse_vector_serializer< SV >::bvector_type_const_ptr

Definition at line 164 of file bmsparsevec_serial.h.

◆ bvector_type_ptr

template<typename SV>
typedef bvector_type* bm::sparse_vector_serializer< SV >::bvector_type_ptr

Definition at line 165 of file bmsparsevec_serial.h.

◆ remap_matrix_type

template<typename SV>
typedef SV::remap_matrix_type bm::sparse_vector_serializer< SV >::remap_matrix_type
protected

Definition at line 257 of file bmsparsevec_serial.h.

◆ serializer_type

template<typename SV>
typedef bm::serializer<bvector_type> bm::sparse_vector_serializer< SV >::serializer_type
protected

Definition at line 263 of file bmsparsevec_serial.h.

◆ size_type

template<typename SV>
typedef SV::size_type bm::sparse_vector_serializer< SV >::size_type

Definition at line 167 of file bmsparsevec_serial.h.

◆ u32_vector_type

template<typename SV>
typedef bm::heap_vector<unsigned, alloc_type, true> bm::sparse_vector_serializer< SV >::u32_vector_type
protected

Definition at line 262 of file bmsparsevec_serial.h.

◆ value_type

template<typename SV>
typedef SV::value_type bm::sparse_vector_serializer< SV >::value_type

Definition at line 166 of file bmsparsevec_serial.h.

Constructor & Destructor Documentation

◆ sparse_vector_serializer()

template<typename SV >
bm::sparse_vector_serializer< SV >::sparse_vector_serializer ( )

Member Function Documentation

◆ build_plane_digest()

template<typename SV >
void bm::sparse_vector_serializer< SV >::build_plane_digest ( bvector_type digest_bv,
const SV &  sv 
)
staticprotected

Definition at line 853 of file bmsparsevec_serial.h.

Referenced by bm::sparse_vector_serializer< SV >::serialize().

◆ build_xor_ref_vector()

template<typename SV >
void bm::sparse_vector_serializer< SV >::build_xor_ref_vector ( const SV &  sv)
protected

◆ disable_xor_compression()

template<typename SV>
void bm::sparse_vector_serializer< SV >::disable_xor_compression ( )
inline

Disable XOR compression on serialization.

Examples:
svsample02.cpp.

Definition at line 205 of file bmsparsevec_serial.h.

References BMNOEXCEPT.

Referenced by SDemo2().

◆ enable_xor_compression()

template<typename SV>
void bm::sparse_vector_serializer< SV >::enable_xor_compression ( )
inline

Enable XOR compression on vector serialization.

See also
set_xor_ref
disable_xor_compression
Examples:
svsample02.cpp.

Definition at line 199 of file bmsparsevec_serial.h.

Referenced by SDemo2().

◆ encode_remap_matrix()

template<typename SV >
void bm::sparse_vector_serializer< SV >::encode_remap_matrix ( bm::encoder enc,
const SV &  sv 
)
protected

◆ get_bv_serializer()

template<typename SV>
bm::serializer<bvector_type>& bm::sparse_vector_serializer< SV >::get_bv_serializer ( )
inline

Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings.

See also
bm::serializer::set_compression_level

Definition at line 247 of file bmsparsevec_serial.h.

◆ is_xor_ref()

template<typename SV>
bool bm::sparse_vector_serializer< SV >::is_xor_ref ( ) const
inline

Returns the XOR reference compression status (enabled/disabled)

Examples:
svsample02.cpp.

Definition at line 228 of file bmsparsevec_serial.h.

References bm::serialize().

Referenced by SDemo2(), and bm::sparse_vector_serializer< SV >::serialize().

◆ serialize()

template<typename SV >
void bm::sparse_vector_serializer< SV >::serialize ( const SV &  sv,
sparse_vector_serial_layout< SV > &  sv_layout 
)

Serialize sparse vector into a memory buffer(s) structure.

Parameters
sv- sparse vector to serialize
sv_layout- buffer structure to keep the result as defined in bm::serialization_flags
Examples:
rscsample05.cpp, and svsample02.cpp.

Definition at line 870 of file bmsparsevec_serial.h.

References bm::serializer< BV >::allow_stat_reset(), bm::bit_out< TEncoder >::bic_encode_u32_cm(), BM_ASSERT, bm::sparse_vector_serializer< SV >::build_plane_digest(), bm::sparse_vector_serializer< SV >::build_xor_ref_vector(), bm::sparse_vector_serializer< SV >::bv_ref_, bm::sparse_vector_serializer< SV >::bv_ref_ptr_, bm::sparse_vector_serializer< SV >::bvs_, bm::globals< T >::byte_order(), bm::sparse_vector_serial_layout< SV >::capacity(), bm::sparse_vector_serializer< SV >::encode_remap_matrix(), bm::bv_ref_vector< BV >::find_bv(), bm::sparse_vector_serial_layout< SV >::get_plain(), bm::id_max32, bm::sparse_vector_serializer< SV >::is_xor_ref(), bm::bv_ref_vector< BV >::not_found(), bm::sparse_vector_serializer< SV >::plane_digest_buf_, bm::sparse_vector_serializer< SV >::plane_digest_bv_, bm::sparse_vector_serializer< SV >::plane_off_vect_, bm::encoder::put_32(), bm::encoder::put_64(), bm::encoder::put_8(), bm::sparse_vector_serial_layout< SV >::reserve(), bm::serializer< BV >::reset_compression_stats(), bm::sparse_vector_serial_layout< SV >::resize(), bm::serializer< BV >::serialize(), bm::serializer< BV >::set_curr_ref_idx(), bm::sparse_vector_serial_layout< SV >::set_plain(), bm::serializer< BV >::set_ref_vectors(), and bm::encoder::size().

Referenced by main(), SDemo2(), serialize_df0(), serialize_df2(), and bm::sparse_vector_serialize().

◆ set_bookmarks()

template<typename SV>
void bm::sparse_vector_serializer< SV >::set_bookmarks ( bool  enable,
unsigned  bm_interval = 256 
)
inline

Add skip-markers for faster range deserialization.

Parameters
enable- TRUE searilization will add bookmark codes
bm_interval- bookmark interval in (number of blocks) (suggested between 4 and 512) smaller interval means more bookmarks added to the skip list thus more increasing the BLOB size

Definition at line 190 of file bmsparsevec_serial.h.

Referenced by main().

◆ set_xor_ref() [1/2]

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_xor_ref ( bool  is_enabled)

Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector.

Default: disabled Reference bit-vectors from the sparse vector itself

Examples:
rscsample05.cpp.

Definition at line 752 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::bv_ref_ptr_, and bm::sparse_vector_serializer< SV >::is_xor_ref_.

Referenced by serialize_df0(), and serialize_df2().

◆ set_xor_ref() [2/2]

template<typename SV >
void bm::sparse_vector_serializer< SV >::set_xor_ref ( const bv_ref_vector_type bv_ref_ptr)

Enable external XOR serialization via external reference vectors (data frame ref.

vector). This method is useful when we serialize a group of related sparse vectors which benefits from the XOR referencial compression

Parameters
bv_ref_ptr- external reference vector if NULL - resets the use of reference vector

Definition at line 742 of file bmsparsevec_serial.h.

References bm::sparse_vector_serializer< SV >::bv_ref_ptr_, and bm::sparse_vector_serializer< SV >::is_xor_ref_.

Field Documentation

◆ bv_ref_

template<typename SV>
bv_ref_vector_type bm::sparse_vector_serializer< SV >::bv_ref_
protected

◆ bv_ref_ptr_

template<typename SV>
const bv_ref_vector_type* bm::sparse_vector_serializer< SV >::bv_ref_ptr_
protected

◆ bvs_

template<typename SV>
bm::serializer<bvector_type> bm::sparse_vector_serializer< SV >::bvs_
protected

◆ is_xor_ref_

template<typename SV>
bool bm::sparse_vector_serializer< SV >::is_xor_ref_
protected

◆ plane_digest_buf_

template<typename SV>
buffer_type bm::sparse_vector_serializer< SV >::plane_digest_buf_
protected

serialization buf

Definition at line 273 of file bmsparsevec_serial.h.

Referenced by bm::sparse_vector_serializer< SV >::serialize().

◆ plane_digest_bv_

template<typename SV>
bvector_type bm::sparse_vector_serializer< SV >::plane_digest_bv_
protected

bv.digest of bit-planes

Definition at line 272 of file bmsparsevec_serial.h.

Referenced by bm::sparse_vector_serializer< SV >::serialize().

◆ plane_off_vect_

template<typename SV>
u32_vector_type bm::sparse_vector_serializer< SV >::plane_off_vect_
protected

Definition at line 274 of file bmsparsevec_serial.h.

Referenced by bm::sparse_vector_serializer< SV >::serialize().

◆ remap_rlen_vect_

template<typename SV>
u32_vector_type bm::sparse_vector_serializer< SV >::remap_rlen_vect_
protected

The documentation for this class was generated from the following file: