BitMagic-C++
|
#include <bmsparsevec_serial.h>
Public Types | |
typedef SV::bvector_type | bvector_type |
typedef const bvector_type * | bvector_type_const_ptr |
typedef bvector_type * | bvector_type_ptr |
typedef SV::value_type | value_type |
typedef SV::size_type | size_type |
typedef bvector_type::allocator_type | alloc_type |
typedef alloc_type::allocator_pool_type | allocator_pool_type |
typedef bm::serializer< bvector_type >::bv_ref_vector_type | bv_ref_vector_type |
typedef bm::serializer< bvector_type >::xor_sim_model_type | xor_sim_model_type |
Public Member Functions | |
sparse_vector_serializer () | |
More... | |
Compression settings <br> | |
void | set_bookmarks (bool enable, unsigned bm_interval=256) BMNOEXCEPT |
Add skip-markers for faster range deserialization. More... | |
void | enable_xor_compression () BMNOEXCEPT |
Enable XOR compression on vector serialization. More... | |
void | disable_xor_compression () BMNOEXCEPT |
Disable XOR compression on serialization. More... | |
void | set_xor_ref (bool is_enabled) BMNOEXCEPT |
Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector. More... | |
void | set_xor_ref (const bv_ref_vector_type *bv_ref_ptr) BMNOEXCEPT |
Enable external XOR serialization via external reference vectors (data frame ref. More... | |
void | compute_sim_model (xor_sim_model_type &sim_model, const bv_ref_vector_type &ref_vect, const bm::xor_sim_params ¶ms) |
Calculate XOR similarity model for ref_vector refernece vector must be associated before. More... | |
void | set_sim_model (const xor_sim_model_type *sim_model) BMNOEXCEPT |
Attach serizalizer to a pre-computed similarity model. More... | |
bool | is_xor_ref () const BMNOEXCEPT |
Returns the XOR reference compression status (enabled/disabled) More... | |
Serialization <br> | |
void | serialize (const SV &sv, sparse_vector_serial_layout< SV > &sv_layout) |
Serialize sparse vector into a memory buffer(s) structure. More... | |
bm::serializer< bvector_type > & | get_bv_serializer () BMNOEXCEPT |
Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings. More... | |
Protected Types | |
typedef SV::remap_matrix_type | remap_matrix_type |
typedef bm::heap_vector< unsigned, alloc_type, true > | u32_vector_type |
typedef bm::serializer< bvector_type > | serializer_type |
typedef serializer_type::buffer | buffer_type |
Protected Member Functions | |
void | build_xor_ref_vector (const SV &sv) |
void | encode_remap_matrix (bm::encoder &enc, const SV &sv) |
serialize the remap matrix used for SV encoding More... | |
Static Protected Member Functions | |
static void | build_plane_digest (bvector_type &digest_bv, const SV &sv) |
Protected Attributes | |
bm::serializer< bvector_type > | bvs_ |
More... | |
bvector_type | plane_digest_bv_ |
bv.digest of bit-planes More... | |
buffer_type | plane_digest_buf_ |
serialization buf More... | |
u32_vector_type | plane_off_vect_ |
u32_vector_type | remap_rlen_vect_ |
bool | is_xor_ref_ |
More... | |
bv_ref_vector_type | bv_ref_ |
xor_sim_model_type | sim_model_ |
const bv_ref_vector_type * | bv_ref_ptr_ |
const xor_sim_model_type * | sim_model_ptr_ |
\brief Serialize sparse vector into a memory buffer(s) structure
Serialization format:
| HEADER | BIT-VECTORS ... | REMAP_MATRIX
BYTE+BYTE: Magic-signature 'BM' or 'BC' (c-compressed) BYTE : Byte order ( 0 - Big Endian, 1 - Little Endian) { BYTE : Number of Bit-vector planes (total) (non-zero when < 255 planes) | BYTE: zero - flag of large plane matrix INT64: Nnmber of bit-vector planes } INT64: Vector size INT64: Offset of plane 0 from the header start (value 0 means plane is empty) INT64: Offset of plane 1 from ... INT32: reserved
Based on current bit-vector serialization
Remap Matrix: SubHeader | Matrix BLOB
sub-header: BYTE: 'R' (remapping) or 'N' (no remapping) N - means no other info is saved on the stream INT64: remap matrix size
Definition at line 175 of file bmsparsevec_serial.h.
void bm::sparse_vector_serializer< SV >::compute_sim_model | ( | xor_sim_model_type & | sim_model, |
const bv_ref_vector_type & | ref_vect, | ||
const bm::xor_sim_params & | params | ||
) |
Calculate XOR similarity model for ref_vector refernece vector must be associated before.
Definition at line 801 of file bmsparsevec_serial.h.
Referenced by serialize_df2().
|
inline |
Disable XOR compression on serialization.
Definition at line 221 of file bmsparsevec_serial.h.
References bm::sparse_vector_serializer< SV >::set_xor_ref().
Referenced by SDemo2().
|
inline |
Enable XOR compression on vector serialization.
Definition at line 215 of file bmsparsevec_serial.h.
References bm::sparse_vector_serializer< SV >::set_xor_ref().
Referenced by SDemo2().
|
protected |
serialize the remap matrix used for SV encoding
Definition at line 830 of file bmsparsevec_serial.h.
References BM_ASSERT, BMRESTRICT, bm::count_nz(), bm::bit_out< TEncoder >::gamma(), bm::encoder::memcpy(), bm::encoder::put_16(), bm::encoder::put_32(), bm::encoder::put_64(), and bm::encoder::put_8().
|
inline |
Get access to the underlying bit-vector serializer This access can be used to fine tune compression settings.
Definition at line 281 of file bmsparsevec_serial.h.
References bm::sparse_vector_serializer< SV >::bvs_.
|
inline |
Returns the XOR reference compression status (enabled/disabled)
Definition at line 260 of file bmsparsevec_serial.h.
References bm::sparse_vector_serializer< SV >::is_xor_ref_.
Referenced by SDemo2().
void bm::sparse_vector_serializer< SV >::serialize | ( | const SV & | sv, |
sparse_vector_serial_layout< SV > & | sv_layout | ||
) |
Serialize sparse vector into a memory buffer(s) structure.
sv | - sparse vector to serialize |
sv_layout | - buffer structure to keep the result as defined in bm::serialization_flags |
Definition at line 929 of file bmsparsevec_serial.h.
References bm::bit_out< TEncoder >::bic_encode_u32_cm(), BM_ASSERT, bm::globals< T >::byte_order(), bm::sparse_vector_serial_layout< SV >::capacity(), bm::sparse_vector_serial_layout< SV >::get_plane(), bm::id_max32, bm::encoder::put_32(), bm::encoder::put_64(), bm::encoder::put_8(), bm::sparse_vector_serial_layout< SV >::reserve(), bm::sparse_vector_serial_layout< SV >::resize(), bm::sparse_vector_serial_layout< SV >::resize_slices(), bm::sparse_vector_serial_layout< SV >::set_plane(), and bm::encoder::size().
Referenced by main(), SDemo2(), serialize_df0(), serialize_df2(), and bm::sparse_vector_serialize().
|
inline |
Add skip-markers for faster range deserialization.
enable | - TRUE searilization will add bookmark codes |
bm_interval | - bookmark interval in (number of blocks) (suggested between 4 and 512) smaller interval means more bookmarks added to the skip list thus more increasing the BLOB size |
Definition at line 207 of file bmsparsevec_serial.h.
References bm::sparse_vector_serializer< SV >::bvs_, and bm::serializer< BV >::set_bookmarks().
Referenced by main().
void bm::sparse_vector_serializer< SV >::set_sim_model | ( | const xor_sim_model_type * | sim_model | ) |
Attach serizalizer to a pre-computed similarity model.
sim_model | - pointer to external computed model |
Definition at line 812 of file bmsparsevec_serial.h.
Referenced by serialize_df2().
void bm::sparse_vector_serializer< SV >::set_xor_ref | ( | bool | is_enabled | ) |
Turn ON and OFF XOR compression of sparse vectors Enables XOR reference compression for the sparse vector.
Default: disabled Reference bit-vectors from the sparse vector itself
Definition at line 792 of file bmsparsevec_serial.h.
Referenced by bm::sparse_vector_serializer< SV >::disable_xor_compression(), bm::sparse_vector_serializer< SV >::enable_xor_compression(), serialize_df0(), and serialize_df2().
void bm::sparse_vector_serializer< SV >::set_xor_ref | ( | const bv_ref_vector_type * | bv_ref_ptr | ) |
Enable external XOR serialization via external reference vectors (data frame ref.
vector). This method is useful when we serialize a group of related sparse vectors which benefits from the XOR referencial compression
bv_ref_ptr | - external reference vector if NULL - resets the use of reference vector |
Definition at line 781 of file bmsparsevec_serial.h.
|
protected |
Definition at line 306 of file bmsparsevec_serial.h.
Referenced by bm::sparse_vector_serializer< SV >::get_bv_serializer(), bm::sparse_vector_serializer< SV >::set_bookmarks(), and bm::sparse_vector_serializer< SV >::sparse_vector_serializer().
|
protected |
Definition at line 314 of file bmsparsevec_serial.h.
Referenced by bm::sparse_vector_serializer< SV >::is_xor_ref(), and bm::sparse_vector_serializer< SV >::sparse_vector_serializer().
|
protected |
serialization buf
Definition at line 309 of file bmsparsevec_serial.h.
|
protected |
bv.digest of bit-planes
Definition at line 308 of file bmsparsevec_serial.h.