Blogs

Introduction and Setup This blog starts with a naive implementation of matmul in C and optimizes it one step at a time. I am using the following machine with a 10 core CPU (4 Performance cores+6 Efficiency cores). Algorithmic complexity of Matrix Multiplication: Calculating the FLOPs required Matrix multiplication is ubiquitous in many areas of computer sciene. As matrices grow in size, the total FLOPs required to calculate the matmul grow cubically1. Let’s see how. ...

In this short post, I am going to talk about the Switch Transformer paper. Background and Architecture To begin: In Dense Models, each parameter/neuron of the network gets activated during the forward pass. Hence, to be able to derive more performance by means of increasing the model size, more computation is required. Implicitly, more model parameters => more computation performed by a single token. And tangential to this, work by Kaplan et al. (2020) becomes super relevant as they discuss training larger models on comparatively smaller amounts of data as the computationally optimal approach. ...

I think differential privacy is beautiful! Why are we here? Protecting the privacy of data is important and not trivial. To help make sense of things here, the Fundamental Law of Information Recovery becomes useful which states: Overly accurate estimates of too many statistics can completely destroy (data) privacy. Another example that provides a good incentive for why privacy is important is the ability of LLMs to memorize data which is an undesirable outcome as it risks the leak of PII. ...

This piece of mine got featured in the Weights and Biases fully connected blog. It can be found here.