Text data typically have high-frequency words such as "the", "a", and "in": they may even occur billions of times in very large corpora. However, these words often co-occur with many different words...

In programming not words
t can the (high-frequency) word w; be discarded, and the higher the relative frequency of the word, the greater the probability of being discarded. "/>
Extracted text: Text data typically have high-frequency words such as "the", "a", and "in": they may even occur billions of times in very large corpora. However, these words often co-occur with many different words in context windows, providing little useful signals. For instance, consider the word "chip" in a context window: intuitively its co-occurrence with a low-frequency word "intel" is more useful in training than the co-occurrence with a high-frequency word "a". Moreover, training with vast amounts of (high-frequency) words is slow. Thus, when training word embedding models, high- frequency words can be subsampled (Mikolov et al., 2013b). Specifically, each indexed word w; in the dataset will be discarded with probability P(w;) = max 1- (14.3.1) f(w;) where f(ur) is the ratio of the number of words w; to the total number of words in the dataset, and the constant t is a hyperparameter (10-4 in the experiment). We can see that only when the relative frequency f(wi) > t can the (high-frequency) word w; be discarded, and the higher the relative frequency of the word, the greater the probability of being discarded.