Data et al.

You have power over your mind - not outside events. Realize this, and you will find strength. -Marcus Aurelius #100Days

Awkwardness, Microsoft Surface Laptop 4, Airflow & Composer


Sometimes I feel like I need to say something in order for common courtesy to take place; even if I don't have anything to say or don't want to say anything. This can't be at all unusual, right?

Anyway, it seems forcing it ends up with me saying things that make little sense since they're so forced. But to say nothing is often considered rude. So is it better to just say anything and risk just coming off aloof?


Microsoft Surface Laptop 4 was announced today. I impulsively placed an order. I have enjoyed the WSL2 experience when combined with Windows Terminal. Makes developing on Windows machines not a complete pain in my ass.

Why did I get this? I'm currently typing on a bohemoth 17" MSI gaming laptop that gets a solid 3-4 hours of time off charge. And while it kicks ass for gaming, it's sort of a terrible choice for general productivity, browsing, and portability if I don't need the power. I'm looking forward to something I can keep off the cord for hours at a time and charge at night like my phone.

Opted for the AMD 512GB with 16GB RAM.

worked on

More work with GCP's Airflow service, Composer. Getting weird memory limit errors with heavy loads on columnarly wide datasets. Scaled up node memory limits within its Kubernetes cluster and still nothing. Fine for now because I got what I need but part of the fun of BigQuery is analytical datastores with wide columnar dataset!

After messing with that for a bit, ended up creating a few more SQL queries to bring interesting views of Salesforce data into Data Studio.

Note to self: Diagram a set of view schemas so I don't have hundreds of custom SQL queries powering hundreds of reports. Reduce, reuse, recycle.

That's all I think.

Getting started on Cloud Run & Kubernetes

I'm getting more and more familiar with GCP and its capabilites as I deploy applications through things like Cloud Run now. This has been sort of a gateway into me wanting to learn about the underlying infrastructure behind it, which has in turn led me to start learning about Kubernetes setups.

I think one difficult thing about all of these options is that there are so many, that you have to really understand all of them in order to make a decision on which one you should be using.

I think if all you care about is saying, give me this CPU size in this region and serve this Image then Cloud Run is great.

I suppose the only real reason I might want to go through the extra work of managing my own kubernetes clusters would be if I wanted to make it a bit more platform agnostic and not be tied into one provider.

I think I might take one of my Cloud Run jobs and deploy a container to a Kubernetes cluster so I get a bit more end-to-end experience with it vs just sort of checking logs of pods that the infra team has procured and set up on their own clusters.

That's one of the things about data engineering; it's difficult sometimes to tell where infrastructure, devops, and data eng should differentiate and overlap. We're working through it live at work since our structure is sort of maturing over time. So far, so good.

That's all for now.

Easing into natural language processing with Hugging Face Transformers

Advancements in AI have brought a lot of attention to a number of subdomains within this vast field. One interesting one is natural language processing.

What is a Hugging Face Transformer?

Why don’t we let their pretrained models answer this question?

Transformers provides general-purpose architectures for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages . The library currently contains PyTorch, Tensorflow and Flax implementations, pretrained model weights, usage scripts and conversion utilities for the following models .

Not bad AI. Not bad at all. The above quote is what a pretrained model using a summarization pipeline provides when applied to the contents of the Hugging Face Transformers documentation.

Using these pipelines allow pretty much anybody to get started down the road of natural language processing without much insight into the back-end of PyTorch or TensorFlow.

How to use Hugging Face Text Summarization

First you have to install the transformers package for Python.

pip3 install transformers

Once you have this installed it is a simple matter of importing the pipeline, specifying the type of model we want to run; in this case summarization, and then passing it your content to summarize.

from transformers import pipeline
text = "Insert a wall of text here"
summarization = pipeline("summarization")
summary_text = summarization(text)[0]['summary_text']

For beginners and experts

The simplicity of these libraries mean you can get started quickly. You can do a lot out of the gate with these libraries and you’ll quickly notice the limitations of the vanilla models. Don’t get me wrong, they are amazing, but if you want to do fine tuning, expect to get reading on some documentation.

I’d suggest identifying a community contributed model that seems interesting and then reverse engineering that if you want to see how they come together.

Ultimately, I believe Hugging Face brings a democratization of NLP for developers in a sense. It is much easier to apply pretrained models to accomplish common tasks such as sentiment analysis, text summarization, and even question generation!

It also opens up NLP and AI practitioners to get involved by contributing to model building and improving the quality of the output that enthusiasts such as myself can enjoy without pouring through documentation tuning parameters when that isn’t my day job!

Give these transformers and pretrained models a try and let me know what you think! Have you found interesting uses for these on any projects?

Hello world

Standard Notes & Listed as a blog

This is my first post using and Standard Notes.

Let's see how it works!

def hello():
  print("Hello World")

if __name__ == '__main__':

Building an attribution model with markov chains

A short while ago I published a rather technical post on the development of a python-based attribution model that leverages a probabilistic graphical modeling concept known as a Markov chain.

I realize what might serve as better content is actually the motivation behind doing such a thing, as well as providing a clearer understanding of what is going on behind the scenes. So to that end, in this post I'll be describing the basics of the Markov process and why we would want to use it in practice for attribution modeling.

What is a Markov Chain

A Markov chain is a type of probabilistic model. This means that it is a system for representing different states that are connected to each other by probabilities.

The state, in the example of our attribution model, is the channel or tactic that a given user is exposed to (e.g. a nonbrand SEM ad or a Display ad). The question then becomes, given your current state, what is your next most likely state?

Well one way to estimate this would be to get a list of all possible states branching from the state in question and create a conditional probability distribution representing the likelihood of moving from the initial state to each other possible state.

So in practice, this could look like the following:

Let our current state be SEM in a system containing the possible states of SEM, SEO, Display, Affiliate, Conversion, and No Conversion.

After we look at every user path in our dataset we get conditional probabilities that resemble this.

P(SEM | SEM) = .1
P(SEO | SEM) = .2
P(Affiliate | SEM) = .05
P(Display | SEM) = .05
P(Conversion | SEM) = .5
P(No Conversion | SEM) = .1

This can be graphically represented.

Notice how the sum of the probabilities extending from the SEM state equal to one. This is an important property of a Markov process and one that will arise organically if you have engineered your datset properly.

Connect all the nodes

Above we only identified the conditional probabilities for scenario in which our current state was SEM. We now need to go through the same process for every other scenario that is possible to build a networked model that you can follow indefinitely.



Now up to this point I've written a lot about the process of defining and constructing a Markov chain but I think at this point it is helpful to explain why I like these models over standard heuristic based attribution models.

Look again at the fully constructed network we have created, but pay special attention to the outbound Display vectors that I've highlighted in blue below.

According to the data, we have a high likelihood of not converting at about 75% and only a 5% chance of converting the user. However, that user has a 20% probability of going proceeding to SEM as the next step. And SEM has a 50% chance of converting!

This means that when it comes time to do the "attribution" portion of this model, Display is very likely to increase its share of conversions.

Attributing the Conversions

Now that we have constructed the system that represents our user behavior it's time to use it to re-allocate the total number of conversions that occured for a period of time.

What I like to do is take the entire system's probability matrix and simulate thousands of runs through the system that end when our simulated user arrives at either conversion or null. This allows us to use a rather small sample to generalize because we can simulate the random walk through the different stages of our system with our prior understanding of the probability of moving from one stage to the next. Since we pass a probability distribution into the mix we are allowing for a bit more variation in our simulation outcomes.

After getting the conversion rates of the system we can simulate what occurs when we remove channels from the system one by one to understand their overall contribution to the whole.

We do this by calculating the removal effect1 which is defined as the probability of reaching a conversion when a given channel or tactic is removed from the system.

In other words, if we create one new model for each channel where that channel is set to 100% no conversion, we will have a new model that highlights the effect that removing that channel entirely had on the overall system.

Mathematically speaking, we'd be taking the percent difference in the conversion rate of the overall system with a given channel set to NULL against the conversion rate of the overall system. We would do this for each channel. Then we would divide the removal CVR by the sum of all removal CVRs for every channel to get a weighting for each of them so that we could finally then multiply that number by the number of conversions to arrive at the fractionally attributed number of conversions.

If the above paragraph confuses you head over to here and scroll about a third of the way down for a clear removal effect example. I went and made my example system too complicated for me to want to manually write out the the removal effect CVRs.

That's it

Well by now you have a working attribution model that leverages a Markov process for allocating fractions of a conversion to multiple touchpoints! I have also built a proof-of-concept in Python that employs the above methodology to perform markov model based attribution given a set of touchpoints.2

  1. Anderl, Eva and Becker, Ingo and Wangenheim, Florian V. and Schumann, Jan Hendrik, Mapping the Customer Journey: A Graph-Based Framework for Online Attribution Modeling (October 18, 2014). Available at SSRN: or