Pageview times experiment (#69)

* experiment: track pageview times

* strictly respect do not track

* oops

* asdfasdfasdf

* add blogpost

* fix typos oops
This commit is contained in:
Cadey Ratio 2019-08-19 13:48:21 -04:00 committed by GitHub
parent eaca47ba37
commit a6c66568c8
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
9 changed files with 166 additions and 34 deletions

View File

@ -0,0 +1,72 @@
---
title: Pageview Time Experiment
date: 2019-08-19
---
# Pageview Time Experiment
My blog has a lot of content in a lot of diverse categories. In order to help me
decide which kind of content I should publish next, I have created a very
simple method to track pageview time and enabled it for all of my blogposts. I'll
go into detail of how it works and potential risks of it below.
The high level idea is that I want to be able to know what kind of content has
people's attention for the longest amount of time. I am using the time people
have the page open as a particularly terrible proxy for that value. I wanted to
make this data anonymous, simplistic and (reasonably) public.
## How It Works
Here is how it works:
<center>![A diagram on how this works](/static/img/pageview_flowchart.png)</center>
When the page is loaded, a [javascript file records the start time](/static/js/pageview_timer.js).
This then sets a [pagehide handler](https://developer.mozilla.org/en-US/docs/Web/API/Window/pagehide_event)
to send a [navigator beacon](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon)
containing the following data:
- The path of the page being viewed
- The start time
- The end time recorded by the pagehide handler
This information is asynchronously pushed to [`/api/pageview-timer`](https://github.com/Xe/site/blob/91d7214b341088edba7a37a83a753e75ed02d7ad/cmd/site/pageview.go)
and added to an in-memory prometheus histogram. These histograms can be checked at
[`/metrics`](/metrics). This data is not permanently logged.
## Security Concerns
I believe this data is anonymous, simplistic and public for the following reasons:
I believe this data is anonymous because there is no way for me to correlate users
to histogram entries, nor is there a way for me to view all of the raw histogram
entries. This site records the bare minimum for what I need in order to make sure
everything is functioning normally, and all data is stored in ephemeral in-memory
containers as much as possible. This includes any logs that my service produces.
I believe this data is simplistic because it only has a start time, a stop time
and the path that is being looked at. This data doesn't take into account things
like people leaving a page open for hours on end idly, and that could skew the
numbers. The API endpoint is also fairly unprotected, meaning that falsified data
could be submitted to it easily. I think that this is okay though.
I believe this data is public because I have the percentile views of the histograms
present on [`/metrics`](/metrics). I have no reason to hide this data, and I do not
intend to use it for any moneymaking purposes (though I doubt it could be to begin
with).
I fully respect the [do not track](https://allaboutdnt.com) header and flag in browsers.
If [`pageview_timer.js`](/static/js/pageview_timer.js) detects the presence of
do not track in the browser, it stops running immediately and does not set the pagehide
handler. If that somehow fails, the server looks for the presence of the `DNT` header
set to `1` and instantly discards the data and replies with a 404.
Like always, if you have any questions or concerns please reach out to me. I
want to ensure that I am creating useful views into how people use my blog
without violating people's rights to privacy.
I intend to keep this up for at least a few weeks. If it doesn't have any practical
benefit in that timespan, I will disable this and post a follow-up explaining how
I believe it wasn't useful.
Thanks and be well.

View File

@ -223,6 +223,7 @@ func Build() (*Site, error) {
w.Header().Set("Content-Type", "application/xml")
_, _ = smi.WriteTo(w)
})))
s.mux.HandleFunc("/api/pageview-timer", handlePageViewTimer)
return s, nil
}

53
cmd/site/pageview.go Normal file
View File

@ -0,0 +1,53 @@
package main
import (
"encoding/json"
"io/ioutil"
"net/http"
"time"
"github.com/prometheus/client_golang/prometheus"
"within.website/ln"
)
var (
readTimes = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "blogpage_read_times",
Help: "This tracks how much time people spend reading articles on my blog",
}, []string{"path"})
)
func init() {
_ = prometheus.Register(readTimes)
}
func handlePageViewTimer(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("DNT") == "1" {
http.NotFound(w, r)
return
}
data, err := ioutil.ReadAll(r.Body)
if err != nil {
ln.Error(r.Context(), err, ln.Info("while reading data"))
http.Error(w, "oopsie whoopsie uwu", http.StatusInternalServerError)
return
}
r.Body.Close()
type metricsData struct {
Path string `json:"path"`
StartTime time.Time `json:"start_time"`
EndTime time.Time `json:"end_time"`
}
var md metricsData
err = json.Unmarshal(data, &md)
if err != nil {
http.NotFound(w, r)
return
}
diff := md.EndTime.Sub(md.StartTime).Seconds()
readTimes.WithLabelValues(md.Path).Observe(float64(diff))
}

4
go.mod
View File

@ -6,9 +6,7 @@ require (
github.com/kr/pretty v0.1.0 // indirect
github.com/pkg/errors v0.8.1 // indirect
github.com/povilasv/prommod v0.0.12
github.com/prometheus/client_golang v0.9.4
github.com/prometheus/common v0.4.1 // indirect
github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389 // indirect
github.com/prometheus/client_golang v1.0.0
github.com/russross/blackfriday v2.0.0+incompatible
github.com/sebest/xff v0.0.0-20160910043805-6c115e0ffa35
github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect

28
go.sum
View File

@ -1,4 +1,3 @@
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@ -8,14 +7,11 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/celrenheit/sandflake v0.0.0-20190410195419-50a943690bc2 h1:/BpnZPo/sk1vPlt62dLya5KCn7PN9ZBDrpTGlQzgUZI=
github.com/celrenheit/sandflake v0.0.0-20190410195419-50a943690bc2/go.mod h1:7L8gY0+4GYeBc9TvqVuDUq7tXuM6Sj7llnt7HkVwWlQ=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@ -37,40 +33,24 @@ github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/povilasv/prommod v0.0.11 h1:uiCU9z2UpNoMFyd5h3evJB8mTBuT9lZU3CYpMZkkyE0=
github.com/povilasv/prommod v0.0.11/go.mod h1:kMc6cpm22gp7m0cPEFRoRgIzXq75ZIJvNY6GbNu9EJk=
github.com/povilasv/prommod v0.0.12 h1:0bk9QJ7kD6SmSsk9MeHhz5Qe6OpQl11Fvo7cvvmNUQM=
github.com/povilasv/prommod v0.0.12/go.mod h1:GnuK7wLoVBwZXj8bhbJNx/xFSldy7Q49A44RJKNM8XQ=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
github.com/prometheus/client_golang v0.9.3 h1:9iH4JKXLzFbOAdtqv/a+j8aewx2Y8lAjAydhbaScPF8=
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
github.com/prometheus/client_golang v0.9.4 h1:Y8E/JaaPbmFSW2V81Ab/d8yZFYQQGbni1b1jPcG9Y6A=
github.com/prometheus/client_golang v0.9.4/go.mod h1:oCXIBxdI62A4cR6aTRJCgetEjecSIYzOEaeAn4iYEpM=
github.com/prometheus/client_golang v1.0.0 h1:vrDKnkGzuGvhNAL56c7DBz29ZL+KxnoR0x7enabFceM=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/common v0.4.1 h1:K0MGApIoQvMw27RTdJkPbr3JZ7DNbtxQNyi5STVM6Kw=
github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389 h1:F/k2nob1S9M6v5Xkq7KjSTQirOYaYQord0jR4TwyVmY=
github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.2 h1:6LJUbpNm42llc4HRCuvApCSWB/WfhuNo9K98Q9sNGfs=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
github.com/russross/blackfriday v2.0.0+incompatible h1:cBXrhZNUf9C+La9/YpS+UHpUT8YD6Td9ZMSU9APFcsk=
github.com/russross/blackfriday v2.0.0+incompatible/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
github.com/sebest/xff v0.0.0-20160910043805-6c115e0ffa35 h1:eajwn6K3weW5cd1ZXLu2sJ4pvwlBiCWY4uDejOr73gM=
@ -82,7 +62,6 @@ github.com/snabb/diagio v1.0.0 h1:kovhQ1rDXoEbmpf/T5N2sUp2iOdxEg+TcqzbYVHV2V0=
github.com/snabb/diagio v1.0.0/go.mod h1:ZyGaWFhfBVqstGUw6laYetzeTwZ2xxVPqTALx1QQa1w=
github.com/snabb/sitemap v1.0.0 h1:7vJeNPAaaj7fQSRS3WYuJHzUjdnhLdSLLpvVtnhbzC0=
github.com/snabb/sitemap v1.0.0/go.mod h1:Id8uz1+WYdiNmSjEi4BIvL5UwNPYLsTHzRbjmDwNDzA=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
@ -92,14 +71,13 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190522155817-f3200d17e092 h1:4QSRKanuywn15aTZvI/mIDEgPQpswuFndXpOj3rKEco=
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
@ -109,9 +87,5 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
within.website/ln v0.5.2 h1:4pTM2wzpLjeZAputLf2U29HD79vcmdoEI/VPm2QEYgs=
within.website/ln v0.5.2/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE=
within.website/ln v0.6.0 h1:zn8vE9f0biSTNIpHFc7kOPH+SitLsWrU1MfrwMSS4cM=
within.website/ln v0.6.0/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE=
within.website/ln v0.7.0 h1:cZUc53cZF/+hWuEAv1VbqlYJ5czuPFHKfH0hLKmlIUA=
within.website/ln v0.7.0/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE=

View File

@ -3,4 +3,4 @@
set -e
docker build -t xena/site .
exec docker run --rm -itp 5000:5000 -e PORT=5000 xena/site
exec docker run --rm -itp 5030:5000 -e PORT=5000 xena/site

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

View File

@ -0,0 +1,31 @@
/*
Hi,
If you are reading this, you have found this script in the referenced scripts
for pages on this site. I know you're gonna have to take me at my word on this,
but I'm literally using this to collect how much time people spend reading my
webpages. See metrics here: https://christine.website/metrics
If you have the "do not track" setting enabled in your browser, this code will
be ineffectual.
*/
(function() {
let dnt = navigator.doNotTrack;
if (dnt === "1") {
return;
}
let startTime = new Date();
function logTime() {
let stopTime = new Date();
window.navigator.sendBeacon("/api/pageview-timer", JSON.stringify({
"path": window.location.pathname,
"start_time": startTime.toISOString(),
"end_time": stopTime.toISOString()
}));
}
window.addEventListener("pagehide", logTime, false);
})();

View File

@ -18,6 +18,9 @@
<link rel="canonical" href="https://christine.website/{{ .Link }}">
<!-- Metrics -->
<script src="/static/js/pageview_timer.js"></script>
<script type="application/ld+json">
{
"@context": "http://schema.org",
@ -69,10 +72,10 @@ function share_on_mastodon() {
if ( !instance.startsWith("https://") && !instance.startsWith("http://") )
instance = "https://" + instance;
// Get the current page's URL
// get the current page's url
var url = window.location.href;
// Get the page title from the og:title meta tag, if it exists.
// get the page title from the og:title meta tag, if it exists.
var title = document.querySelectorAll('meta[property="og:title"]')[0].getAttribute("content");
// Otherwise, use the <title> tag as the title
@ -98,7 +101,7 @@ function share_on_mastodon() {
// Open a new window at the share location
window.open(mastodon_url, '_blank');
}
}
}
</script>
{{ end }}