ALEXANDRIA, Va., July 7 -- United States Patent no. 12,675,646, issued on July 7, was assigned to S&P Global Inc. (New York).
"Optimal subword tokenization and vocabulary creation" was invented by Craig William Schmidt (Wellesley, Mass.).
According to the abstract* released by the U.S. Patent & Trademark Office: "Subword tokenization is provided. The method comprises receiving a text document comprising n bytes and specifying a maximum token width of L bytes. An initial vocabulary of tokens is defined, wherein the tokens comprise a number of different n-grams of L or less bytes. The document is tokenized with the fewest number of tokens from the vocabulary according to a minimum total weight through a directed acyclic graph comprising nod...