tokenizer.py File Reference

Classes
class	core.tokenizer.Tokenizer
	DVM Token creator. More...
class	core.tokenizer.ContextManager
	Context Manager. More...
class	core.tokenizer.Switcher
	Token Switcher. More...

Namespaces
	core.tokenizer
	DVM token creator.

Variables
tuple	core.tokenizer.log = logging.getLogger(__name__)