Skip to contents

Read a raw WEXTOR server log file

Source: R/read_WEXTOR_log.R
read_WEXTOR_log.Rd

Reads a raw .txt access log downloaded from a WEXTOR study (https://wextor.eu) into a data frame with one row per logged HTTP request, i.e. one row per page or asset served during the study. This can then be brought into a tidy (one participant per row) format using tidy_WEXTOR_log. If you suspect there may be double entries or other issues with the data, it can help to check the original log file. The tidy version also gets a new variable added that flags potential anomalies like a double IP address.

The log format WEXTOR writes is a tab-separated custom Apache LogFormat equivalent to: "%P %\{%m/%d/%y\}t %\{%T\}t %c %h %U %B User-Agent Referer %T %q", with one further, undocumented field appended after %q that appears to be the server's response time in milliseconds. Concretely, every line in the file has 12 tab-separated fields:

  1. process_id – Apache worker/process id (%P)

  2. date – request date, mm/dd/yy

  3. time – request time, HH:MM:SS

  4. connection_status – Apache connection status (%c: "+" keep-alive, "-" closed, "X" aborted)

  5. client_ip – requesting IP address (%h)

  6. url – requested path, host/researcher/study/page (%U)

  7. bytes_sent – response size in bytes (%B)

  8. user_agent – browser/client user-agent string

  9. referer – the URL the request came from

  10. time_taken_s – time to serve the request, in whole seconds (%T)

  11. query_string – the raw GET query string (%q): the WEXTOR variables submitted together with this request

  12. duration_ms – server response time in milliseconds. This field isn't part of WEXTOR's documented log format and its exact meaning is inferred from the data – treat it with mild caution and verify against your own server setup if it matters for your analysis.

date and time are combined into one timestamp column, and url is split into host, study_path and page for convenience. query_string is additionally parsed into individual name/value pairs and kept as a list-column, query – one small tibble per request, one column per WEXTOR variable that was submitted with it.

Note that WEXTOR logs each request's query string as only the variables submitted from the previous page (plus id), not the full cumulative set collected so far – so a single row is not a full participant record. To get one row per participant, use tidy_WEXTOR_log() from this package.

Repeated keys within one query string (which happens for checkbox variables WEXTOR logs once per click) are resolved by keeping the last value, i.e. the final state at submission.

Usage

read_WEXTOR_log(filepath, tz = "UTC", drop_assets = FALSE)

Arguments

filepath

Path to a WEXTOR log .txt file.

tz

Time zone to assign to the parsed timestamp. WEXTOR logs the server's local time, not necessarily UTC. Default "UTC"; set this to your server's actual time zone if absolute times matter.

drop_assets

If TRUE, drop requests for static assets (css/js/images/fonts) and keep only the study's own HTML pages. Default FALSE.

Value

A tibble with one row per logged HTTP request and the columns described above (plus host, study_path, page, timestamp, and the query list-column). Lines that don't split into exactly 12 tab-separated fields (e.g. a truncated trailing line) are dropped with a warning.

Examples


log <- read_WEXTOR_log(path_to_file("fake_wextor_log.txt"))