prometheus on somtochiama

Understanding Native Histograms in Prometheus

somtochionyekwere@gmail.com (Somtochi Onyekwere) — Mon, 06 Apr 2026 15:04:15 +0100

Histogram is a metric type in Prometheus used to track the number (and sums) of observations such as latency and response times. It allows you to answer questions like how many requests were completed under 100ms. In version 2.24.0, Prometheus added support for a new type of histograms called native histogram. This new histogram is similar to the original classic histograms but differ in how bucket boundaries are determined, and how the histogram data structure is represented internally.

Unlike classic histograms, which rely on statically configured bucket boundaries, native histograms do not require predefined boundaries. Instead, they use a schema that determines how bucket boundaries are calculated. This approach allows for a potentially infinite number of buckets while remaining memory-efficient because native histograms are sparse, buckets with zero observations consume almost no space. This blog takes a peek at native histograms.

Bucket boundaries

Histograms works by keeping a count of each observation that is less than or equal to the upper bound of a bucket. Classic histogram are configured by specifying the upper boundary of each bucket. Picking the bucket is left to the user, they pick values that they care about (if you care that requests take less than 10s, you’ll probably have something setup like [0.1, 0.2, 0.5, 1, 2, 5, 10]). All requests taking more than 10s falls into the last bucket whether it takes 10secs or 2 minutes. This works fine for most applications, at ten seconds you are already screaming regardless of how far out the actual numbers are, but if you care about the numbers that don’t fall into the defined buckets, this might be insufficient.

Native histograms have dynamic resolution and hence can produce more accurate calculations, while still being memory efficient because they are sparse - a bucket with zero observation takes up no space. The bucket boundaries of a native histogram is determined by what prometheus calls a schema. The scehma controls by how quickly bucket widths grow from one bucket to the next. You define a number between -4 and 8, and the bucket width of each successive bucket grows exponentially by (2 ^ (2 ^ (-schema)). So if you have a bucket with a schema of 0, the bucket with increases exponentially by 2 (2 ^( 2 ^ -0)) = 2. This means that the upper boundary of each bucket is twice that of the previous bucket.

The higher the schema number the higher the resolution, since the growthnfactor of the bucket width decreases.

Sparseness

Although you theorectically have an infinite amount of buckets, native histograms are sparse. We only keep data for buckets that have seen observations in. Prometheus achieves this by keeping additional information about the layouts of the buckets to easily identify filled buckets. The bucket with the upper boundary of 1 is taken as the zeroth buckets, and negative buckets (that’d record negative observations) are to the left of the zeroth bucket and have negative indexes, while positive bucket are to the left and have +ve indexes. The histogram records spans - continous segments of filled buckets. It has both length (to identify how many buckets are in a span) and offset - its distance from the last span.

Histogram { spans = []Span buckets = []f64}Span { length u64 offset u64}

For example a native histogram, which has no observations in its zeroth bucket, but has observed counts in its first to third bucket, and then more counts in its sixth and seventh buckets can be represented as shown.

Histogram { Spans = []{ Span{offset: 1, length: 3} Span{offset: 2, length: 2} } buckets: [3, 6, 5, 1, 2] }

The first offset of 2 tells us that there are two empty buckets after the first one, then four filled buckets after that, and so forth. The counts of the filled buckets are contained in the buckets field.

Mergability

Additionally, Prometheus provides an option to limit the number of buckets and when the number of buckets exceeds this number, it would automatically merge buckets of the histogram into one with a smaller schema.

Histograms with different schemas can be merged to produce a histogram with a lower schema. Because the bucket widths would differ by a power of 2 between successive schemas we can merge neighbouring buckets to get a new histogram a smaller schema. Take the example histograms below, we can merge the first two buckets for the first histogram ([1->2] with count 3 and [2->4] with count 6) to get one bucket 1->4 with count of 9, then the two histograms are easily mergable.

Quantile estimatation

Quantiles in histogram is commonly used to describe latency or response time behaviours in application e.g to answer questions, like what is the latency of 99th percent of my requests, which is important in many applications. For classic histogram, linear interpolation is used to the estimate the nth percentile. It is assumed that the distribution within the buckets are uniform.

For native histograms, it is assumed that observations within a bucket follow a uniform distribution when the bucket is subdivided exponentially. In other words, if a bucket is split into smaller sub-buckets with exponentially spaced boundaries, the samples are assumed to be evenly distributed across those sub-buckets.

Echoing code comment in prometheus code:

For exponential buckets, we interpolate on a logarithmic scale. On a logarithmic scale,the exponential bucket boundaries (for any schema) become linear (every bucket has the same width).

Exponential interpolation is done by first applying linear interpolation for the exponents of the bucket boundaries, then the final value is obtained by raising 2 to the power of the result.

// code has been simplified for clarity here// the actual code has special hang custom buckets, NaN values and zero bucketsfor it.Next() { // returns the next bucket with its count, lower and upper bound (in increasing order). bucket = it.At() if bucket.Count == 0 { continue } count += bucket.Count // we accumulate the total here if count >= rank { // then check if the rank falls within this bucket break }}logLower := math.Log2(math.Abs(bucket.Lower)) // exponent of lower boundarylogUpper := math.Log2(math.Abs(bucket.Upper)) // exponent of upper boundaryrank = count - rank // how many observations falls within the last bucketfraction := rank / bucket.Count // what fraction of the bucket is thatpower := logLower + (logUpper-logLower)*fraction // linear interpolation on the exponenntsreturn math.Exp2(power), annos // we take a power of two here to get the actual value.

You can look the code here promql/quantile.

I think native histograms can be useful in a lot of cases, it allows use of labels on histograms without blowing up cardinality and finer grained resolution for more accurate measurements. Native histograms can be enabled using the scrape_native_histograms config setting but note that this changes the exposition format to protobuf since text is not yet supported.

References:

Getting accepted into GSoC

somtochionyekwere@gmail.com (Somtochi Onyekwere) — Tue, 01 Jul 2025 00:58:56 +0100

This blog post is about five years too late. I got into Google Summer of Code in 2020 and have been meaning to write one, but I kept putting it off. Since I still get occasional messages from people trying to get in, and the application period is closed, I decided to finally sit down and get it done. Getting into the program quite literally changed my life. I learnt a whole lot and connected with lifelong mentors (hi Leighhhh 💓!). This post is mostly around getting into Google Summer of Code, but the tips are general enough that they apply to other open-source internships, and there are many others like Outreachy, MLH, LFX Mentorship etc.

Maybe an intro on why you should care?

If you are just getting started in tech, have a solid knowledge of one or two programming languages, have built some personal projects, and you are looking for a way to level up your skills and get paid while at it, open-source internships are where it is. You get a chance to contribute to an active project, get feedback and reviews from experienced people and join a great community while getting paid (in $$). There are seasoned engineers from big tech companies and fast-paced startups in these communities, and you also get a chance to connect with them. Each project idea has specific mentors, and you get guidance throughout the internship period.

When I first applied to GSoC in 2019 (yes, I applied twice, it is okay if you don’t get in the first time), I only started to look at organizations during the application period, and just managed to scrape together a proposal. But when the next application season came around, I was ready.

Start early

The most important thing you can do to increase your chances is to start early, like wayy before the application period begins. Get involved and start contributing as soon as you can.Of course, organizations would pick someone who has already been contributing for a while. The person would also have a better proposal because they have more knowledge about the project. You can stop reading right now. That’s it. What follows are just tips to help with decisions, picking a project, etc. But at the core of it is to pick a project that aligns with what you are interested in (language that you know, and doing something you are interested in) and making meaningful contributions early. (Please, please don’t be one of those people who make random, empty contributions that add to maintainer burden; this wouldn’t help AT ALL.)

Pick an organisation

It helps if you are genuinely interested in the project, and it aligns with your experience. It is much easier to be a consistent contributor if the project intrigues you. The Google summer of codede archive page](https://summerofcode.withgoogle.com/archive) has tabs to check organizations in particular areas like Machine Learning.

Take a look at what they do, read through their docs. Some repositories have a CONTRIBUTING.md file that contains information for new contributors. You can join a couple of communities before settling on one, introduce yourself as a new contributor, try to understand what the project is about and ask what are good first steps to take, Most communities are very welcoming, but I tried to look out for more active communities since you’d get responses faster.

You can select two organisations, but I selected just one (Cloud Native Computing Foundation). I found it easier to focus all my efforts on one; most of them are big enough that they have multiple project ideas during GSoC, so you can still submit two or three proposals.

Contribute!

This is where the work lies. It’s not enough to just join these communities. You have to actually contribute. It is okay if it feels overwhelming at first. The project is probably years of work and has evolved - you can’t understand it all within a week. Most organizations have a bunch of repositories, but there’s normally one or two core ones. I’d advise starting with those or whichever piques your interest.

Start small. Some repositories have labelled issues that indicate the level of difficulty. It doesn’t have to be code; you can start with documentation. Go through the docs and demos - you might find something that’s broken or doesn’t work as stated. Add new tests or fix an easy issue that no one has gotten around to. You can also simply ask - ‘ Hi, I am a new contributor and I’m excited to get started, are there any good issues that’d be great to pick?’.

Join the community meetings and try to be a part of the discussion. Ask questions when you are confused. Be open to reviews on your pull request, each code base has its own styles and entrenchment. Don’t get stressed because there are a lot of review comments - just carefully go through them and make the necessary changes.

But don’t get stuck in the small things; the goal over time is for you to progress to handling bigger issues. With time and effort, you’ll become a regular contributor. Check out other people’s pull requests and issues too, don’t be afraid to make suggestions or open new issues that you might have discovered.

Put in maximum effort during the proposal period:

It is possible that the part of the project that you’ve been working on is different from the ones the organization will submit as its project ideas. When you discover this, shift your effort to that area. Start contributing there, join the meetings if it is different from the ones you have been attending. Understand what the project idea is about and what they’re trying to achieve.

Aim to write a very solid proposal. Bring fresh ideas to the table, and clearly break down your plan into achievable milestones over the three-month period. Take a look at past accepted proposals to get some inspiration. My friend created a repository containing proposals from different organizations here.

That’s it! I hope this helps. Best of luck 💫

If you are curious about what I worked on during the GSoC period. Here’s a link to a blog post around it. https://kubernetes.io/blog/2020/09/16/gsoc20-building-operators-for-cluster-addons/

Learning to Touch type

somtochionyekwere@gmail.com (Somtochi Onyekwere) — Sat, 10 Feb 2024 22:51:24 +0100

we are typist first, programmers second - Jeff Atwood

Quick log about how I got to touch typing at 50 wpm from hunt-and-peek in roughly 6 months.

Focus on accuracy and speed will come.
Start with keybr.com until all the letters turn green
- Aim for about 15 - 30mins everyday, also a fun way to fill in your pomodoro breaks
- You don’t need to touch type during your daily work, that might make you frustrated. Just 30 minutes is enough,
- You are just trying to internalize the positions of the keys and get used to it
- One great thing about keybr is that it keeps you on one key until you are able to hit it at a certain speed andaccuracy. That’s why I think one of the best for starting out, even if you already know the position of the keys
- keybr doesn’t show the finger placement on the keyboard, you might have to take a look an image to figure out whichfinger goes with which key.
- Unfortunately, keybr doesn’t include letters, symbols or modifier keys. You might have to supplement withanother tool (typing.com?). I can no longer remember which tool I used. Any tool should be fine. You just need toknow the positions of each key.
When all your letters are green on keybr, Move to monkeytype.com
- My setting for this is 1k (1k most common words in English), numbers, punctuations and 60(secs,length of each typing test, planning to increase it 120secs)
- Aim for >97% accuracy. Worth mentioning again - accuracy over speed
- At the end of each test, you can retry the words that you missed and keep doing this until you hit a 100% accuracy.

Since I got to a reasonable speed and I could touch type for pretty much my daily work.I stopped the daily practice and plateaued at 50 wpm sometime last year. I hope to get back at this year and get to70wpm. If you have any other good tips for going from 50 wpm to 70 wpm. leave a comment!

Bloggers on Blogging

somtochionyekwere@gmail.com (Somtochi Onyekwere) — Mon, 12 Jun 2023 12:31:27 +0100

I have spent a sometime convincing myself to start a blog and in doing that I have stumbled upon good advice onblogging. So I decided to publish a post each week to get my engines started. I am starting off by compiling a list oflinks from people I follow/read their blogs on why you should blog.

Jeff Atwood: “How To Achieve Ultimate Blog Success In One Easy Step”

Jeff writes at https://blog.codinghorror.com which is one of my favourite blogs.

When people ask me for advice on blogging, I always respond with yet another form of the same advicepick a schedule you can live with, and stick to it. Until you do that, none of the other advice I could giveyou will matter. I don’t care if you suck at writing.I don’t care if nobody reads your blog. I don’t careif you have nothing interesting to say. If you candemonstrate a willingness to write, and a desire to keep continually improving your writing, you will eventually be successful.

Ben Kuhn: Writing

Ben Kuhn goes more in-depth on how and why you should blog.

It’s easy to think that you have to put out really “interesting” writing in order for other people to like it.3I think that’s true for going viral, but not true for having a blog that your friends enjoy reading,makes acquaintances feel more positively towards you, etc.….As mentioned above, an important first step here is to lower your bar for what’s worth writing about.

Alex Guezy: Why You Should Start A Blog Right Now

This fact is very frequently lost when discussing writing: writing not only helps you to understandwhat’s going on and to crystallize your thoughts, it actually makes you think of new ideas and comeup with solutions to your problems.

Steve Yegge: You should write blogs

I’ve talked with a lot of people who are reluctant to write blogs. Everyone offers pretty much the same reasons:they’re too busy, or they’re afraid to put something on “permanent public record”, or they think nobody willread their blog, or they think blogging is narcissistic. Or they’re worried that they either don’t have anythinggood to say, or they won’t say it very well.
I’m here to tell you that none of these reasons should stop you from writing in your blog.

Julia Evans: Write aboout what you’ve struggled with

This is more about getting a sense of what to blog about, especially for programmers.Notice the crease, parts that don’t make sense, delve deeper into them and write about them.

The process I use for a lot of my blog posts is:
Struggle with something (usually computer-related)
Eventually (days or months or years later), figure out how to solve some of the problems I had
Write a blog post about what helped me

David Perrell: Why you should write:

Perrell’s not really into tech but has some good writing tips. He also runs an online writing school Write of Passage:

Words are the atomic unit of the internet.With the stroke of a pen, you can build your network, improve your thinking, and create opportunities for yourself.

Seth Godin: Write something

Seth keeping it short and sweet, as always.

There’s no such thing as writer’s block. There’s simply a fear of bad writing.Do enough bad writing and some good writing is bound to show up.

I am mostly writing this for myself, in case I occasionally need a reminder.The core of the advice on these links is just to get started! More specifically, start writing(and publishing)at a particular cadence online.

You know a lot of cool stuff, you should share them :).

Welcome to my blog!

somtochionyekwere@gmail.com (Somtochi Onyekwere) — Sat, 15 Aug 2020 12:31:27 +0100

My Lord, If I may?

Welcome, wanderers, to my corner of the internet. However you may have stumbled into this rough path, it matters not. This is my space and you are definitely welcome. This place houses my writings - technical or non-technical, rants, and what not.

My name is Somto. You may have heard of me.