Steem Tag Covariance

I looked at the Steem tags used in May 2018 using a covariance matrix. Covariance measures how closely two random variables match each other's values. Here, the random variables are binary variables (0 or 1) indicating whether or not a tag is present, and our observations are 1172186 top-level discussions. To make the analysis manageable, I limited it to tags with 50 posts or more, which produced a set of 5180 tags.

The sample covariance can be calculated from this set of observations with some linear algebra. The observations can be represented as a 1172186x5180 matrix --- fortunately a very sparse one. Each row is one discussion, with '1's set in the columns corresponding to the tags attached to that discussion. Using the built-in numpy covariance function would convert this to a dense matrix and thus run out of memory. But I found a calculation on Stackoverflow that preserves sparseness, though the resulting covariance matrix is not itself sparse:

    # Compute the covariance matrix
    rowsum = A.sum(1)
    centering = rowsum.dot(rowsum.T.conjugate()) / n
    C = (A.dot(A.T.conjugate()) - centering) / (n - 1)

I thought I was going to be able to produce a nice heatmap of the results, but unfortunately most of the values are so close to zero that it just looks like a single color! The covariance tells us which tags are most likely to be used together; that is, which tags predict the presence of other tags. The i,j entry in the matrix calculated above indicates the covariance between tags i and j.

The highest covariances in this data set are the tags that are most often used together:

But which tags have the lowest covariance? A selection of tags that are most negatively covariant against other tags:

That is, a discussion that uses #cryptocurrency is somewhat less likely to also use #life or #photography. Unsurprisingly, apps are common here--- nobody uses both #busy and #esteem at the same time.

What about our friend #steemstem?

With statistical tools like this, we can start to extract some semantic information, as well as better understand how people use Steem.

Tags are constrained in a way that is hard to model: a discussion can normally have only up to five of them. (Using the API directly appears to allow more.) This means that the behavior of tags is not fully captured by their pairwise behavior which we have been looking at here. So while I am trying to model them using a Bayesian approach, I have no idea what the underlying "expected" distribution is when we choose five different tags according to their relative independent distributions. My question asking for help on Quora has thus far gone unanswered: https://www.quora.com/I-have-a-deck-consisting-of-n_k-copies-of-the-card-k-How-do-I-calculate-the-probability-of-drawing-a-five-card-hand-containing-two-specified-cards-a-and-b-What-if-a-hand-may-have-no-duplicates