By Merlin Carter
Who owns the task of user tracking? Is it Product Management? Is it Marketing? or is it the Data team? The answer depends on who you ask. But it’s usually the front-end engineers who have to implement it. Yet, with so many stakeholders, things can get complicated. That’s why I decided to chat with our front-end team lead, Michael Lande, about how he approaches the challenges involved in user tracking.
The catalyst for our conversation was a document that he’d started titled User Analytics Engineering. He was motivated to capture the patterns, risks, and approaches involved in implementing user tracking. It’s a multifaceted topic that often falls under the radar of some front-end engineers, so he wanted to give his team a handy reference. Michael and I discussed his experience with user tracking and how it gave him the idea for this kind of document.
You can read our conversation below. I’ve also included a brief summary of Michael’s User Analytics Engineering document after the transcript of our conversation.
Merlin: Today. I am very excited because we have a new face on tech insights, and that wonderful new face is Michael Lande, who leads our front-end engineering team. Welcome to the show, Michael.
Michael: Thanks for having me, Merlin, nice to be here.
What do you do at Project A?
Merlin: Great. So up until now, Tech Insights has been a series of conversations between myself and our CTO, Stefan Schulze, but I realize that there are plenty of other people at Project A who also have a lot of useful insights to offer (including you, Michael).. and since it’s the first time that we are doing a Tech Insights together, perhaps you can just briefly describe what it is you do as front-end team lead at Project A.
Michael: Sure. so my name is Michael, and I’ve been leading one of our software engineering teams at Project A for the last almost two years. And, what we do, as a team, we focus on frontend, which means that we do frontend-related projects with our portfolio companies. And the types of projects that we do are varied.
Mostly I would say that one big chunk of the work we do is joining teams. We join product engineering teams as developers. We integrate ourselves into the team and either add developer resources or certain specific expertise that we have to help them develop their products or scale.
These are usually longer-running projects, and we also do shorter-term or more-scoped projects that are more, maybe, consultancy-like. We do some audits, uh, performance audits or code audits. We do, sometimes, really just knowledge exchange that helps people see what direction they want to take in a certain business decision.
So all kinds of things that can help our portfolio with their technical challenges… frontend related.
Why did you put together a briefing on user analytics engineering?
Merlin: Gotcha. Okay. Thanks. So the catalyst for this discussion is an internal document that you started called “User Analytics Engineering”. And in that document, you talk about user tracking from a business and a technical perspective, and you describe the challenges and pitfalls in dealing with different stakeholders and their tracking requirements when you are building a front-end application. So, first off, can you tell me you decided to create this document and who exactly did you write it for?
“I wanted to document the commen patterns that I’ve observed when working with user tracking.”
Michael: Yeah, that’s a good question. I think, to get an idea of what this whole thing is about. So the broader context is that we do a lot of knowledge-sharing within Project A. This is part of what we do. I mean, I guess this is also part of why we’re here. We also do it a lot internally, specifically within our frontend team, we’ve recently had a lot of knowledge exchange in relation to building some knowledge base into common topics that we deal with as front developers. That’s the broader context of what this document is part of. But specifically, the reason I chose this topic was my recent, and also not so recent experience with it.
I have noticed that the topic of tracking is a topic that arises almost in any web application that I was involved in, whether it arises later on when the product is already there or during the development process. And I’ve noticed common patterns every time we talk about tracking. So the idea here was to try to describe them in a very broad sense to give an overview from the engineering team’s perspective. So, of course, analytics and tracking is a broad topic and has a lot of stakeholders. So this wasn’t really supposed to give a deep dive into what this is actually used for the business it was really more like an overview of why it’s there. And then, from our engineering team’s perspective, what is our role in it?
So it’s a kind of checklist for your team?
Merlin: Gotcha. So, if I could summarize that, it’s sort of like an internal checklist for you and your team to sort of go through and make sure you’ve thought of everything when you start a project.
Michael: Yeah, I mean, I’m a big fan of checklists. I wouldn’t call it so much of a checklist because I think it needs a bit more refinement to be an operational checklist. It’s still, at this moment, more of like an overview that you might want to read as a front-end engineer and get an idea of things to look out for.
When you start a new project, do you also get detailed instructions on how things should be tracked?
Merlin: Gotcha. All right. So I wanna jump into the process of starting a new project and either building a new frontend application or working on an existing one. So, let’s just take something where you build something from scratch.
So you obviously would get the design mock-ups, the interaction design, and what have you. So you’ve got a good idea of how it should work, but at that stage, at the requirements gathering stage, do you also get detailed instructions on how it should be tracked and the analytics that people expect to get out of it?
“Usually what I observe is that… tracking or analytics gets pushed to the back”
Michael: So the answer to that would be, in an ideal world, yes. In practice, mostly no. This is actually also one of the catalysts, I guess for me writing this document is that I recently had an experience where we were developing with one of our portfolio companies a very complex web application. And I realized somewhere during the development process that there is so much going on.
It’s a very user-rich application… it is very, very certain that the company would want to know a lot about what’s going on, about how people are using the application. Specifically for improvements to the product. And, of course, also for other purposes that have to do with, maybe, more general business goals. And I was not the only one to think about that, but the group of us thought about that and brought that topic forward.
And I think, and this is something I was very happy with… since we did that in advance, we came to the situation where we already talked about our tracking, about analytics goals… while we were developing, and then we embedded it into the development process. But usually, what I observe is actually not like that.
Usually, what I observe is that… tracking or analytics gets pushed to the back. And very often, it’s not always, of course, some companies are very data-driven, but very often, uh, companies are rushing to get a product out, which is understandable.
And once they get the product out, they realize that they’re flying blind, so to speak, because they are making many assumptions, and now they don’t really have a way to validate those assumptions. And that’s where tracking becomes very urgent. And that’s also when it becomes trickier to do and some tech debt if you’re into it. But if you integrate it into the development process in advance, it just becomes, as you described it, part of the requirements.
And that is, for me, kind of the goal. So I would just say that if you’re already in a situation where you’re adding tracking into an existing product, that’s okay to do it after the fact. And then, say, moving forward, start integrating it into the tickets themselves as part of the acceptance criteria.
Then, then it forces you, for every feature, to think not only about what the feature is but what do you want to know about this? And also, it forces you to do that before you even ship it.
How easy is it to integrate tracking after you’ve released?
Merlin: Yeah, that’s a very good process suggestion. I remember working a long time ago as a web designer. You know you get people from different departments saying, can you put this tracking code on there? And you got that tracking code in there. And all of a sudden, your web pages slowly loaded with all these mysterious tracking codes sitting in the script tag or whatever, but for an actual application, are there design decisions that you can make that make it easier for people to add tracking later on if they do forget to think of tracking at the beginning? How easy is it to integrate tracking after you’ve released?
Michael: Yeah. so, that, of course, depends on what you mean by tracking and how your application looks like, but let’s try to narrow it down.
Because tracking is a broad topic, I mean, both across the stack of the front end or back end, and both from marketing to analytics, like in the domain, you’re trying to use it for, but if we’re talking about, like UI tracking or web UI tracking. You want to analyze the activity of users when they use your user interface.
If that is your goal, usually you have some mixture or combination of some custom tracking that you’re doing, which means you integrate some clients, some tool that you can log events to… events with some properties on what’s going on. So, for example, the user submitted an order.
So, you can log that intent to submit an order, uh, to whatever tracking tool you have. And you also have these fully automatic scripts that you would embed. And they really do this. What we used to call pixel tracking. Right. So they really just listen to every event on your browser and push these automatic events when the user clicks any button or moves around in any place.
These are things like, for example, GTM, you can use it for that. And other, of course, a million tools. So, to answer your question about hard it is, I would say that generally, it shouldn’t the hat hard because, again, if we’re talking about user interface tracking, you’re trying to hook into some user activity that you were reacting to anyways, in code and as a side effect of that, you’re also trying to just push data to some service, right.
So, for example, when the user clicks on something, you are already reacting to that, right? That’s a feature of the app. You just need to hook into that and push something, log something to a service.
So theoretically, hypothetically, that shouldn’t be complicated. But, of course, that’s not always the case. So there are many cases where this can be more complicated. One case can be… if you want to track things that maybe you’re not explicitly writing in your own code. So maybe you want to track things that are more native behavior of the system.
So, for example, if you want to track things that are related to scrolling… what the user is observing scrolling, you might not have any code in the app that deals with that because it’s just the native feature of the browser, but maybe someone wants to know something about what the user is observing, in the scroll.
So that’s where you have to not only log the events but also write specific code that listens to the event. so I think one complexity is… how much you already have the listening part. and then another complexity is just the design of your software, right? So if your software is designed in a way that makes it hard to hook into that moment for whatever reason, maybe it was written fast, and you didn’t create it in a modular way.
Then it might be just hard, but that’s as hard as adding any other feature. It’s not specific to analytics.
How do you manage different stakeholders’ tracking requirements? Are there overlaps?
Merlin: Yeah, I guess it does add extra work overhead if people don’t think of it early enough. because it’s just extra coding work.
And I was also thinking about how the requirements of these different stakeholders overlap. So there’s… obviously different people, different teams who want to track.
So, you know, I think about all these tools like Hotjar and Amplitude and Mixpanel, and, you know, they, they all designed to track the analytics interactions, hotspots, where people move their mouse, which is kind of nothing to do with the functionality of the product. And then there might be more marketing-oriented things like, I don’t know….stuff, by HubSpot, where the people open some content from the app where the people do conversions, and then there’s your own team, which is probably interested in speed… i.e., do different components download fast enough?… and various aspects of the web app performance.
So how do you manage all of these different requirements and other overlaps—are you tracking the same thing in different tools?
Michael: Yeah. I agree that this is a very bloated space, right? There are many tools, and there are many different interests that are sometimes conflicting within the same organization. Right. I would say that one way to reason about this, and I think we hinted at that before, is to treat this like a feature, right?
Or at least as like part of a feature. So it’s not a feature by itself, but it is part of the feature development. And once it’s part of a ticket or part of the acceptance criteria for fulfilling the ticket’s requirements, then you can budget for it the same way you budget for any other feature.
For example, if you have it not as an afterthought but as part of your process, and when you look at a task, and you analyze and try to estimate how complex it is, then if the tracking for that specific task is very hard, that will add complexity to that task.
And then, the team will be forced to deal with it in the same way it deals with any other complexity. So you will have to prioritize. So you either have to remove some of the functionality you want, or you would need to prioritize it or accept that it will take longer or push it down into the future and not do it now. The same way you would do with any feature that you know, you might analyze and, in a way, look at its value for money, right?
How much value it’s giving for you as a company or how much value is given for the users compared to how complex it is and how risky it is. Um, that’s the general approach. Right.
But there are also things that are trickier with the tools themselves. So, for example, a lot of these tools, I haven’t worked with many of them, but I worked with a few, and one of the problems is that they’re very generic. If you, for example, don’t want to do any custom tracking, right? You just want to integrate something.
So usually, you have some stakeholder that comes and says: “Hey, look, I have this cool library”. All you need to do is embed this code, and you will just give me tracking, right?
But of course, that way of thinking would have worked many years ago, maybe… when on the one hand, compliance was not such a big deal… on the other hand, web applications were not so complex. There were really just pages that were loading. What you see today is when you really just tried to embed these things, they don’t just work.
So, for example, in single-page applications that don’t really have full page loads all the time, you always have a problem with page views. You always have to find a custom solution. I haven’t seen tools really dealing with that perfectly yet. There have been improvements, but, for example, sometimes they make assumptions in these pixel tracking, right? That a page is relatively stable until it’s moved around or navigated, where, in fact, in a single-page application, for example, pages are extremely dynamic.
You cannot assume that, for example, a certain CSS class that selects a certain HTML element be there second later or after this interaction. Everything is changing all the time. So, these tools, I think, are limited. Although I do admit that they’re clearly improving. I mean, because they realize that these things are changing.
So there’s no one answer, right? If you need to figure it out, look at it as the same budget you look at it as features.
Do you include the testing of tracking functionality when estimating test effort?
Merlin: Yeah. I like it when you talk about budgets and estimations, we haven’t even talked about testing, which is something else you raise in the document. It’s extra work and complexity to integrate these different tracking libraries and tools. But you also have to test that on top of just the functionality. So you have a team to test the functionality and the tool, and the web app, sorry, you also need to test whether the stats are coming through and whether they’re being tracked properly. So that adds extra time to the testing estimations, right?
“Yes, you should add a certain layer of analytics testing. Usually that would involve the stakeholder of that tracking”
Michael: Absolutely. So, when you do tracking, of course, you need to test the integration of the tracking. So, there are a few aspects of this:
One, you need to first test the integration with the tool itself. Usually, if you embed an SDK or some client that you want to see first that it’s working, that’s kind of a one-time effort. But then, for every event that you add, every new thing needs to test… if it’s custom, right? Not if it’s like this tool that brings everything out of the box. So, for example, you want to track a specific event that is specific to your domain, and you write explicit code to log that event because that’s a certain type.
Then you need to test that yet. So here as well, in the same way, you embed the analytics requirements into your feature development process. You should embed some kind of analytics testing mechanism in your QA process, whatever that might be. People do QA in different ways, but there is some level of quality assurance that companies use to give themselves confidence that they’re shipping what they want to ship.
so whatever that may be, you should add a certain layer of analytics testing. Usually, that would involve the stakeholder of that tracking, or maybe someone, some person from the data team that is in charge of the actual data, uh, consumption on the other end of the pipeline. So what we did, for example, in this last project where we added tracking, was to just create a few points on how we think is a good way to test.
We agreed on this with the data team and engineering team. We said, okay, this is how we give ourselves confidence that we are tracking correctly. So that involved a few things. First of all, we have a template on how we define requirements because requirements for tracking can be very ambiguous in their wording.
And if they are ambiguous, you might be tracking, not what you think you’re tracking. To give an example: So a data analyst might think that we’re tracking the click of a button, and the engineer might implement that we’re tracking the user reaching the next screen.
So it really depends on the case, but this subtlety can be very impactful. Because these things could be vastly different in terms of metrics. And you might make business decisions based on one, but you’re actually not measuring them.
So, first of all, we aligned on how we define requirements. We had strict language on how we define what an event really is from the user’s perspective and what the intent we are logging is, then we said, how are we going to test this? So we basically aligned on when this thing is done, we assign the ticket to the data analyst, and the data analyst uses this debug tool that we have on the tool itself to test it end to end. So to actually play around in the staging environment and see that they’re getting the events on the other end with the correct properties. And then internally, we also have unit tests that cover the logging itself.
So this is the end-to-end test that is done manually. I’m sure you can automate it somehow, but I didn’t get to that level of sophistication yet, but you also internally write tests to make sure that… given certain user activity, you are trying to log this thing… and then in the end-to-end test, you check that the thing actually arrived in the other [end].
Should engineers get the product teams and data teams into a room and make sure that they’re aware of all this?
Merlin: Gotcha. But suppose that I’m a new front-end engineer joining a small startup, like one of our portfolio companies, and I’m one of the first hires or maybe a second hire. So. Would it be a recommendation if I’m working on a new feature to get the product teams and the data engineering teams into a room and just make sure that they’re aware of all this, it’s probably not something they would think of on their own.
Right. So, it’s good for the frontend engineer to be proactive and get these requirements or these processes in place.
“Yes, it‘s in our best interest to be proactive. It’s also probably a part of what I would call professional integrity”
Michael: Yes. No, I agree completely with that, it is in our best interest to be proactive. It’s also probably a part of what I would call professional integrity that if we know this, we should flag it. I think. We should. Yes, exactly. Do that way, that puts the right people in the room first and foremost, or a virtual room.
You first talk about this thing generally, really shortly, let’s say, do we want to do it? How do we want to do it? What tool do we want to use? What are our constraints? and then, from there, you start to refine the process on the details. Like the things, we discussed now, like how are you going to test it? How are you going to define it? Are you going to make it part of the tickets?
But first, align on what you really want to achieve and do that relatively early, it doesn’t need to be the first thing you do, but sometimes during the development of a new product, or sometime during the development of the new feature of an existing product, this is something you should definitely push for.
And this is even more important… if you mentioned it before. And I think we didn’t really go into it in detail. But it’s not only about product analytics, right? There’s also… for example, marketing and interests involved. And, often, we have cases where marketing wants to embed… for example, something like GTM, which is a tool that empowers them and is very powerful for marketing people to use.
However, there are a lot of risks involved. Now, if you don’t talk about this together with everyone and clearly state: “what are the risks? how do we mitigate them?”… then you will end up either with frustrated people because the engineering team is just constantly going to block the marketing team or complain, and there’s going to be this lack of trust, or you’re just going to get the risks, blowing up in your face, right?
Like, for example, the website performance is impacted, or even the website is broken completely, which I’ve personally observed with a bad GTM script.
And that doesn’t mean that we don’t need to use [GTM] because these tools exist for a reason. And I think empowering stakeholders to do things on the website that provides them with insight without needing engineering resources is a good thing, but we still need to maintain the same level of quality assurance for any kind of code that goes into your website.
And that is true. Both for code that goes for the features that developers write and do things like code reviews or pair programming or tests, and it’s as true for code that goes through a tool like GTM. It doesn’t mean that that code is less “real”…
Merlin: GTM… that’s Google tag manager, right?
Michael: Google tag manager. Exactly. Yes. So this is also something to align. This is also something we did recently. So, for example, we agreed with the stakeholders of Google tag manager. We came up with a process on how we ensure the quality of stuff that goes into that tool because, in the end, it’s going to run on the same web app.
But we agreed on some form of code review for every custom script that is added… and also some performance benchmarking that people do… to make sure we’re not deteriorating in performance.
Merlin Carter: Great. Well, thank you very much for that. We‘re pushing the boundaries of the average Tech Insights. This is, this is just the start. I think we should talk more about this and have more content about this. Because this is really a very valuable reference, and I hope we get to talk again. But thank you for taking the time out for this Tech Insights, and I hope to see you again.
Michael Lande: Thanks a lot, Merlin, it was very interesting. And see you again soon.
A Brief Summary of User Analytics Engineering
User tracking is essential for any online business. It provides business stakeholders with:
- Insights on user journeys — For example, insights on parts of your UI that need improvement (with a high user dropout rate or low usage).
- Metrics on critical payment funnels — For example, to identify a high-impact product and allocate more resources to it while de-prioritizing unimpactful markets and channels.
- Metrics on onboarding funnels —For example, these metrics might indicate that you should invest more in marketing platform “X“ because it converts better.
What’s the role of engineering?
In terms of technical investment, there are usually two teams who have a vested interest in getting tracking set up properly:
- The product engineering team implements logic to track user interactions.
These interactions could be direct UI interactions between the user and the frontend or user-initiated transactions between the frontend and the back end.
These two streams of data are often tagged with some form of shared identity (session IDs, user IDs), which allows the data team to match frontend and backend events. - The data team handles data enrichment, transformation, persistence, and visualization of data. They are often responsible for producing reporting dashboards for commercial teams such as marketing and sales.
With unified tracking data about events in the front and back end, the data team can then answer questions such as:
— “What proportion of intents to purchase a product actually succeed?”
— “How many products does one user typically purchase?”
Challenges
Tracking is typically implemented by listening to activity on the UI and logging corresponding user events to an analytics service (with some context parameters).
However, there are some common challenges and pitfalls that you should account for in the product development process:
- The Interaction Logic Challenge
In some cases, you might be asked to track something that you don’t normally log — e.g., a “list_scroll_end” event when the user has reached the end of a listing page.
In this case, it’s extra work because we can’t recycle any existing logging logic. You have to build your own. So you should find out ahead of time if this is part of the tracking requirements. - The Compliance Challenge
Tracking often involves user consent management to ensure that it is only done to the extent legally approved by the user. This usually means managing some global “consent state”, which is manipulated by some “consent UI” (like a cookie banner). The user analytics module must be able to access the consent state and connect or disconnect tracking accordingly. This must be thoroughly tested, which can add time to t - The Testing Challenge:
As the end-to-end functionality involves several systems, as well as several data transformations, testing the implementation of user analytics isn’t trivial. You need to test:
— The UI layer in isolation to ensure interactions are being logged locally.
— The integration between the UI layer and the analytics platform to ensure logged events make it into the analytics database.
— The end-to-end data pipeline to ensure that user tracking data in the UI layer is matched with events in the back end and shows up correctly in the analytics tool. - The “Page View” Challenge
Page views are key to observing user journeys, as all other events are usually analyzed in the context of a page view. But apps with client-side navigation and rendering pose a challenge in two parts:
— Defining what constitutes a “page view”, especially when it comes to modal dialogs and tabbed pages.
— Accurately logging a page view for single-page applications. To understand this problem in detail, see the Google Analytics guidelines for SPAs.
The guiding principle for tackling this is the user journey. Pages should be defined and tracked in a way that allows stakeholders to observe a user’s progress through the app in rich detail. This is a challenge shared by stakeholders, data teams, and product engineering teams. Consequently, those teams need to discuss these tracking challenges together.