diff --git a/blog/pageview-time-experiment-2019-08-19.markdown b/blog/pageview-time-experiment-2019-08-19.markdown new file mode 100644 index 0000000..ecb3b8d --- /dev/null +++ b/blog/pageview-time-experiment-2019-08-19.markdown @@ -0,0 +1,72 @@ +--- +title: Pageview Time Experiment +date: 2019-08-19 +--- + +# Pageview Time Experiment + +My blog has a lot of content in a lot of diverse categories. In order to help me +decide which kind of content I should publish next, I have created a very +simple method to track pageview time and enabled it for all of my blogposts. I'll +go into detail of how it works and potential risks of it below. + +The high level idea is that I want to be able to know what kind of content has +people's attention for the longest amount of time. I am using the time people +have the page open as a particularly terrible proxy for that value. I wanted to +make this data anonymous, simplistic and (reasonably) public. + +## How It Works + +Here is how it works: + +
![A diagram on how this works](/static/img/pageview_flowchart.png)
+ +When the page is loaded, a [javascript file records the start time](/static/js/pageview_timer.js). +This then sets a [pagehide handler](https://developer.mozilla.org/en-US/docs/Web/API/Window/pagehide_event) +to send a [navigator beacon](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon) +containing the following data: + +- The path of the page being viewed +- The start time +- The end time recorded by the pagehide handler + +This information is asynchronously pushed to [`/api/pageview-timer`](https://github.com/Xe/site/blob/91d7214b341088edba7a37a83a753e75ed02d7ad/cmd/site/pageview.go) +and added to an in-memory prometheus histogram. These histograms can be checked at +[`/metrics`](/metrics). This data is not permanently logged. + +## Security Concerns + +I believe this data is anonymous, simplistic and public for the following reasons: + +I believe this data is anonymous because there is no way for me to correlate users +to histogram entries, nor is there a way for me to view all of the raw histogram +entries. This site records the bare minimum for what I need in order to make sure +everything is functioning normally, and all data is stored in ephemeral in-memory +containers as much as possible. This includes any logs that my service produces. + +I believe this data is simplistic because it only has a start time, a stop time +and the path that is being looked at. This data doesn't take into account things +like people leaving a page open for hours on end idly, and that could skew the +numbers. The API endpoint is also fairly unprotected, meaning that falsified data +could be submitted to it easily. I think that this is okay though. + +I believe this data is public because I have the percentile views of the histograms +present on [`/metrics`](/metrics). I have no reason to hide this data, and I do not +intend to use it for any moneymaking purposes (though I doubt it could be to begin +with). + +I fully respect the [do not track](https://allaboutdnt.com) header and flag in browsers. +If [`pageview_timer.js`](/static/js/pageview_timer.js) detects the presence of +do not track in the browser, it stops running immediately and does not set the pagehide +handler. If that somehow fails, the server looks for the presence of the `DNT` header +set to `1` and instantly discards the data and replies with a 404. + +Like always, if you have any questions or concerns please reach out to me. I +want to ensure that I am creating useful views into how people use my blog +without violating people's rights to privacy. + +I intend to keep this up for at least a few weeks. If it doesn't have any practical +benefit in that timespan, I will disable this and post a follow-up explaining how +I believe it wasn't useful. + +Thanks and be well. diff --git a/cmd/site/main.go b/cmd/site/main.go index b7774b1..04d8bc4 100644 --- a/cmd/site/main.go +++ b/cmd/site/main.go @@ -223,6 +223,7 @@ func Build() (*Site, error) { w.Header().Set("Content-Type", "application/xml") _, _ = smi.WriteTo(w) }))) + s.mux.HandleFunc("/api/pageview-timer", handlePageViewTimer) return s, nil } diff --git a/cmd/site/pageview.go b/cmd/site/pageview.go new file mode 100644 index 0000000..5154659 --- /dev/null +++ b/cmd/site/pageview.go @@ -0,0 +1,53 @@ +package main + +import ( + "encoding/json" + "io/ioutil" + "net/http" + "time" + + "github.com/prometheus/client_golang/prometheus" + "within.website/ln" +) + +var ( + readTimes = prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Name: "blogpage_read_times", + Help: "This tracks how much time people spend reading articles on my blog", + }, []string{"path"}) +) + +func init() { + _ = prometheus.Register(readTimes) +} + +func handlePageViewTimer(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("DNT") == "1" { + http.NotFound(w, r) + return + } + + data, err := ioutil.ReadAll(r.Body) + if err != nil { + ln.Error(r.Context(), err, ln.Info("while reading data")) + http.Error(w, "oopsie whoopsie uwu", http.StatusInternalServerError) + return + } + r.Body.Close() + + type metricsData struct { + Path string `json:"path"` + StartTime time.Time `json:"start_time"` + EndTime time.Time `json:"end_time"` + } + var md metricsData + err = json.Unmarshal(data, &md) + if err != nil { + http.NotFound(w, r) + return + } + + diff := md.EndTime.Sub(md.StartTime).Seconds() + + readTimes.WithLabelValues(md.Path).Observe(float64(diff)) +} diff --git a/go.mod b/go.mod index eb15ac7..6179c85 100644 --- a/go.mod +++ b/go.mod @@ -6,9 +6,7 @@ require ( github.com/kr/pretty v0.1.0 // indirect github.com/pkg/errors v0.8.1 // indirect github.com/povilasv/prommod v0.0.12 - github.com/prometheus/client_golang v0.9.4 - github.com/prometheus/common v0.4.1 // indirect - github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389 // indirect + github.com/prometheus/client_golang v1.0.0 github.com/russross/blackfriday v2.0.0+incompatible github.com/sebest/xff v0.0.0-20160910043805-6c115e0ffa35 github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect diff --git a/go.sum b/go.sum index 037ce43..356c5d9 100644 --- a/go.sum +++ b/go.sum @@ -1,4 +1,3 @@ -github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= @@ -8,14 +7,11 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/celrenheit/sandflake v0.0.0-20190410195419-50a943690bc2 h1:/BpnZPo/sk1vPlt62dLya5KCn7PN9ZBDrpTGlQzgUZI= github.com/celrenheit/sandflake v0.0.0-20190410195419-50a943690bc2/go.mod h1:7L8gY0+4GYeBc9TvqVuDUq7tXuM6Sj7llnt7HkVwWlQ= -github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no= github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as= github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE= -github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk= github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY= github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= @@ -37,40 +33,24 @@ github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U= -github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/povilasv/prommod v0.0.11 h1:uiCU9z2UpNoMFyd5h3evJB8mTBuT9lZU3CYpMZkkyE0= -github.com/povilasv/prommod v0.0.11/go.mod h1:kMc6cpm22gp7m0cPEFRoRgIzXq75ZIJvNY6GbNu9EJk= github.com/povilasv/prommod v0.0.12 h1:0bk9QJ7kD6SmSsk9MeHhz5Qe6OpQl11Fvo7cvvmNUQM= github.com/povilasv/prommod v0.0.12/go.mod h1:GnuK7wLoVBwZXj8bhbJNx/xFSldy7Q49A44RJKNM8XQ= github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw= -github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM= -github.com/prometheus/client_golang v0.9.3 h1:9iH4JKXLzFbOAdtqv/a+j8aewx2Y8lAjAydhbaScPF8= -github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= -github.com/prometheus/client_golang v0.9.4 h1:Y8E/JaaPbmFSW2V81Ab/d8yZFYQQGbni1b1jPcG9Y6A= -github.com/prometheus/client_golang v0.9.4/go.mod h1:oCXIBxdI62A4cR6aTRJCgetEjecSIYzOEaeAn4iYEpM= github.com/prometheus/client_golang v1.0.0 h1:vrDKnkGzuGvhNAL56c7DBz29ZL+KxnoR0x7enabFceM= github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 h1:S/YWwWx/RA8rT8tKFRuGUZhuA90OyIBpPCXkcbwU8DE= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= -github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= -github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= -github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/common v0.4.1 h1:K0MGApIoQvMw27RTdJkPbr3JZ7DNbtxQNyi5STVM6Kw= github.com/prometheus/common v0.4.1/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= -github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= -github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= -github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389 h1:F/k2nob1S9M6v5Xkq7KjSTQirOYaYQord0jR4TwyVmY= -github.com/prometheus/procfs v0.0.0-20190523193104-a7aeb8df3389/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= github.com/prometheus/procfs v0.0.2 h1:6LJUbpNm42llc4HRCuvApCSWB/WfhuNo9K98Q9sNGfs= github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA= -github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU= github.com/russross/blackfriday v2.0.0+incompatible h1:cBXrhZNUf9C+La9/YpS+UHpUT8YD6Td9ZMSU9APFcsk= github.com/russross/blackfriday v2.0.0+incompatible/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= github.com/sebest/xff v0.0.0-20160910043805-6c115e0ffa35 h1:eajwn6K3weW5cd1ZXLu2sJ4pvwlBiCWY4uDejOr73gM= @@ -82,7 +62,6 @@ github.com/snabb/diagio v1.0.0 h1:kovhQ1rDXoEbmpf/T5N2sUp2iOdxEg+TcqzbYVHV2V0= github.com/snabb/diagio v1.0.0/go.mod h1:ZyGaWFhfBVqstGUw6laYetzeTwZ2xxVPqTALx1QQa1w= github.com/snabb/sitemap v1.0.0 h1:7vJeNPAaaj7fQSRS3WYuJHzUjdnhLdSLLpvVtnhbzC0= github.com/snabb/sitemap v1.0.0/go.mod h1:Id8uz1+WYdiNmSjEi4BIvL5UwNPYLsTHzRbjmDwNDzA= -github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= @@ -92,14 +71,13 @@ golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20190522155817-f3200d17e092 h1:4QSRKanuywn15aTZvI/mIDEgPQpswuFndXpOj3rKEco= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= @@ -109,9 +87,5 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= -within.website/ln v0.5.2 h1:4pTM2wzpLjeZAputLf2U29HD79vcmdoEI/VPm2QEYgs= -within.website/ln v0.5.2/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE= -within.website/ln v0.6.0 h1:zn8vE9f0biSTNIpHFc7kOPH+SitLsWrU1MfrwMSS4cM= -within.website/ln v0.6.0/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE= within.website/ln v0.7.0 h1:cZUc53cZF/+hWuEAv1VbqlYJ5czuPFHKfH0hLKmlIUA= within.website/ln v0.7.0/go.mod h1:ifURKqsCJekcsdUE+hyCdcuhQqQ+9v9DfA++ZqYxZFE= diff --git a/scripts/docker.sh b/scripts/docker.sh index 8eb3c20..bb9d91a 100755 --- a/scripts/docker.sh +++ b/scripts/docker.sh @@ -3,4 +3,4 @@ set -e docker build -t xena/site . -exec docker run --rm -itp 5000:5000 -e PORT=5000 xena/site +exec docker run --rm -itp 5030:5000 -e PORT=5000 xena/site diff --git a/static/img/pageview_flowchart.png b/static/img/pageview_flowchart.png new file mode 100644 index 0000000..99b1ee0 Binary files /dev/null and b/static/img/pageview_flowchart.png differ diff --git a/static/js/pageview_timer.js b/static/js/pageview_timer.js new file mode 100644 index 0000000..2761396 --- /dev/null +++ b/static/js/pageview_timer.js @@ -0,0 +1,31 @@ +/* + Hi, + + If you are reading this, you have found this script in the referenced scripts + for pages on this site. I know you're gonna have to take me at my word on this, + but I'm literally using this to collect how much time people spend reading my + webpages. See metrics here: https://christine.website/metrics + + If you have the "do not track" setting enabled in your browser, this code will + be ineffectual. +*/ + +(function() { + let dnt = navigator.doNotTrack; + if (dnt === "1") { + return; + } + + let startTime = new Date(); + + function logTime() { + let stopTime = new Date(); + window.navigator.sendBeacon("/api/pageview-timer", JSON.stringify({ + "path": window.location.pathname, + "start_time": startTime.toISOString(), + "end_time": stopTime.toISOString() + })); + } + + window.addEventListener("pagehide", logTime, false); +})(); diff --git a/templates/blogpost.html b/templates/blogpost.html index c5a61c9..61a4b38 100644 --- a/templates/blogpost.html +++ b/templates/blogpost.html @@ -18,6 +18,9 @@ + + + {{ end }}