zdict.h File Reference

#include <stddef.h>

Include dependency graph for zdict.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes
struct	ZDICT_params_t

Macros
#define	ZSTD_ZDICT_H
#define	ZDICTLIB_VISIBLE
#define	ZDICTLIB_HIDDEN
#define	ZDICTLIB_API   ZDICTLIB_VISIBLE

Functions
ZDICTLIB_API size_t	ZDICT_trainFromBuffer (void *dictBuffer, size_t dictBufferCapacity, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples)
ZDICTLIB_API size_t	ZDICT_finalizeDictionary (void *dstDictBuffer, size_t maxDictSize, const void *dictContent, size_t dictContentSize, const void *samplesBuffer, const size_t *samplesSizes, unsigned nbSamples, ZDICT_params_t parameters)
ZDICTLIB_API unsigned	ZDICT_getDictID (const void *dictBuffer, size_t dictSize)
ZDICTLIB_API size_t	ZDICT_getDictHeaderSize (const void *dictBuffer, size_t dictSize)
ZDICTLIB_API unsigned	ZDICT_isError (size_t errorCode)
ZDICTLIB_API const char *	ZDICT_getErrorName (size_t errorCode)

Macro Definition Documentation

ZDICTLIB_API

#define ZDICTLIB_API   ZDICTLIB_VISIBLE

Definition at line 47 of file zdict.h.

ZDICTLIB_HIDDEN

#define ZDICTLIB_HIDDEN

Definition at line 38 of file zdict.h.

ZDICTLIB_VISIBLE

#define ZDICTLIB_VISIBLE

Definition at line 30 of file zdict.h.

ZSTD_ZDICT_H

#define ZSTD_ZDICT_H

Definition at line 16 of file zdict.h.

Function Documentation

ZDICT_finalizeDictionary()

ZDICTLIB_API size_t ZDICT_finalizeDictionary	(	void *	dstDictBuffer,
		size_t	maxDictSize,
		const void *	dictContent,
		size_t	dictContentSize,
		const void *	samplesBuffer,
		const size_t *	samplesSizes,
		unsigned	nbSamples,
		ZDICT_params_t	parameters
	)

ZDICT_finalizeDictionary(): Given a custom content as a basis for dictionary, and a set of samples, finalize dictionary by adding headers and statistics according to the zstd dictionary format.

Samples must be stored concatenated in a flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample in order. The samples are used to construct the statistics, so they should be representative of what you will compress with this dictionary.

The compression level can be set in parameters. You should pass the compression level you expect to use in production. The statistics for each compression level differ, so tuning the dictionary for the compression level can help quite a bit.

You can set an explicit dictionary ID in parameters, or allow us to pick a random dictionary ID for you, but we can't guarantee no collisions.

The dstDictBuffer and the dictContent may overlap, and the content will be appended to the end of the header. If the header + the content doesn't fit in maxDictSize the beginning of the content is truncated to make room, since it is presumed that the most profitable content is at the end of the dictionary, since that is the cheapest to reference.

maxDictSize must be >= max(dictContentSize, ZSTD_DICTSIZE_MIN).

Returns

: size of dictionary stored into dstDictBuffer (<= maxDictSize), or an error code, which can be tested by ZDICT_isError(). Note: ZDICT_finalizeDictionary() will push notifications into stderr if instructed to, using notificationLevel>0. NOTE: This function currently may fail in several edge cases including:

• Not enough samples
• Samples are uncompressible
• Samples are all exactly the same

Definition at line 858 of file zdict.c.

Here is the call graph for this function:

Here is the caller graph for this function:

ZDICT_getDictHeaderSize()

ZDICTLIB_API size_t ZDICT_getDictHeaderSize	(	const void *	dictBuffer,
		size_t	dictSize
	)

Definition at line 109 of file zdict.c.

Here is the call graph for this function:

ZDICT_getDictID()

ZDICTLIB_API unsigned ZDICT_getDictID	(	const void *	dictBuffer,
		size_t	dictSize
	)

extracts dictID;

Returns

zero if error (not a valid dictionary)

Definition at line 102 of file zdict.c.

Here is the call graph for this function:

ZDICT_getErrorName()

ZDICTLIB_API const char * ZDICT_getErrorName

(

size_t

errorCode

)

Definition at line 100 of file zdict.c.

Here is the call graph for this function:

ZDICT_isError()

ZDICTLIB_API unsigned ZDICT_isError

(

size_t

errorCode

)

Definition at line 98 of file zdict.c.

Here is the call graph for this function:

Here is the caller graph for this function:

ZDICT_trainFromBuffer()

ZDICTLIB_API size_t ZDICT_trainFromBuffer	(	void *	dictBuffer,
		size_t	dictBufferCapacity,
		const void *	samplesBuffer,
		const size_t *	samplesSizes,
		unsigned	nbSamples
	)

ZDICT_trainFromBuffer(): Train a dictionary from an array of samples. Redirect towards ZDICT_optimizeTrainFromBuffer_fastCover() single-threaded, with d=8, steps=4, f=20, and accel=1. Samples must be stored concatenated in a single flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample, in order. The resulting dictionary will be saved into dictBuffer.

Returns

: size of dictionary stored into dictBuffer (<= dictBufferCapacity) or an error code, which can be tested with ZDICT_isError(). Note: Dictionary training will fail if there are not enough samples to construct a dictionary, or if most of the samples are too small (< 8 bytes being the lower limit). If dictionary training fails, you should use zstd without a dictionary, as the dictionary would've been ineffective anyways. If you believe your samples would benefit from a dictionary please open an issue with details, and we can look into it. Note: ZDICT_trainFromBuffer()'s memory usage is about 6 MB. Tips: In general, a reasonable dictionary has a size of ~ 100 KB. It's possible to select smaller or larger size, just by specifying dictBufferCapacity. In general, it's recommended to provide a few thousands samples, though this can vary a lot. It's recommended that total size of all samples be about ~x100 times the target size of dictionary.

Definition at line 1107 of file zdict.c.

Here is the call graph for this function: