Almost every organization that operates over extended periods of time accumulates legacy infrastructure, and Statistics Canada is certainly no stranger to this phenomenon. With some effort one can find workarounds for many of these, but there is one bit of data infrastructure that has been particularly infuriating: The Beyond 20/20 IVT file format.
TL;DR
The canivt package reads Statistics Canada’s Beyond 20/20 IVT files in R and converts them to CSV or parquet. Point it at a StatCan catalogue number, or the id of a cross tabulation hosted on Borealis, and it downloads the file, parses it, and hands back a connection to a parquet database.
That matters because IVT is a closed format adopted by (the census division of) Statistics Canada in the early 1990s, and a large corpus of census data is still locked in it — older census data, where it is the only way to get the table, and custom tabulations, which for most of that period were released this way and no other. Getting at any of it has meant installing proprietary Windows-only software and exporting to CSV by hand: cumbersome, full of idiosyncratic manual choices, prone to silent data loss on low-memory computers, and fatal to any automated or reproducible workflow starting from authoritative StatCan data.
# A tibble: 330 × 6
geo_label geo_name geo_uid Single Years of Age …¹ `Sex (3)` value
<chr> <chr> <chr> <fct> <fct> <dbl>
1 British Columbia | … British… 59 Total - Age Groups Total - … 3.28e6
2 British Columbia | … British… 59 Total - Age Groups Male 1.63e6
3 British Columbia | … British… 59 Total - Age Groups Female 1.66e6
4 British Columbia | … British… 59 0-4 years Total - … 2.21e5
5 British Columbia | … British… 59 0-4 years Male 1.13e5
6 British Columbia | … British… 59 0-4 years Female 1.08e5
7 British Columbia | … British… 59 Under 1 Total - … 4.41e4
8 British Columbia | … British… 59 Under 1 Male 2.24e4
9 British Columbia | … British… 59 Under 1 Female 2.17e4
10 British Columbia | … British… 59 1 Total - … 4.49e4
# ℹ 320 more rows
# ℹ abbreviated name: ¹`Single Years of Age (110)`
The key contribution is reverse-engineering the IVT file format itself; the package is the usable end of that.
What is Beyond 20/20
Beyond 20/20 is a catchall for either an undocumented data format, “Beyond 20/20 IVT tables”, or the proprietary Windows-only software that reads (free) or writes (paid) this data. The Beyond 20/20 software is designed to handle cross-tabulations where data can be sliced along several dimensions in a table-based interface.
Beyond 20/20 is a product of its time. The file format is trying to save space while not having access to modern compression. The browser gives a table view that allows for iterative filtering and pivoting. At the time data work was done in dinosaur systems like SAS and usage of these tools was low. Beyond 20/20 offered an intuitive Excel-like experience that broadened the user base of the data. The trade-off was a closed data format and loss of control over the data format, as well as the tools to extract data for analysis.
When the census division settled on IVT in the early 90s, StatCan data was itself closed up — available only after paying a fee, with no right to share or redistribute it. Against that backdrop, releasing it in a closed format was not the contradiction it looks like today. What has not aged well is everything since. Statistics Canada switched to an open data model in 2012, and more than a decade later a good chunk of their data is still available only as Beyond 20/20 IVT. That is an ongoing failure of data governance, and it severely limits the usefulness of the data.
As we have noted before in relation to the canpumf package (von Bergmann 2026), at least the data is available, and where it matters enough — as in this case — people outside the organization can put in the work and turn it into open data by breaking down the barriers to accessing it.
What to do about Beyond 20/20
For standard census products IVT is not always the only option. There is SDMX for data from the early 2000s onward, and before that an HTML interface where the data can be viewed one slice at a time and scraped with enough effort and patience. Both are annoying and time-consuming, but they exist. For custom tabulations they generally do not: IVT has traditionally been the only way StatCan would distribute them, with no alternative format at all.
So researchers wanting the data either bite the bullet — find a Windows machine, fiddle with the Beyond 20/20 browser, extract what they need by hand — or, in a lot of cases, simply don’t bother, and bump the research that would have needed Beyond 20/20 data to some other time. It is hard to know how much work never happens for this reason, which is its own kind of cost. Either way, fully reproducible research on Beyond 20/20 data is impossible, because the manual extraction process is highly idiosyncratic.
Using the browser is painful in ways anyone with even passing exposure to modern data practices will recognize, and extracting a dataset in bulk is surprisingly difficult. In many cases it’s impossible to extract the geographic data, the geo codes, and the quality flags at the same time. And it is a manual step that breaks reproducibility: chances are high that the next person extracts the data differently. Modern data practices require a more principled and programmatic way to get at the data and the metadata. Which would be straightforward, except that the format is proprietary and undocumented. Additionally, the export of large tables can cut short due to memory constraints, leading to silent data loss at export.
Reverse engineering Beyond 20/20 IVT tables
This is the tricky part. We did give this a try a couple of years back but gave up. Recently we had another need to look at IVT tables and decided to give it another go, this time bringing our sidekick Claude along to help out with the more repetitive and brute-force pieces of cracking the encoding.
We know that the format was formalized sometime in the very early 1990s. That takes more modern data compression and encoding algorithms out of the picture and lets us focus on fairly simple ways someone might have coded something like this around 1990. From some of the performance issues we can also conclude that the encoding is likely not terribly sophisticated. This gives us a starting point and some hope that decoding this is possible.
The other reason for why this might be doable is that we have data to validate against, either the scraped HTML for older data, or the SDMX or CSV for modern data. In essence this is attempting a Known-Plaintext Attack, trying to break the IVT cipher.
A firm rule kept the whole thing honest: parsing has to be driven by metadata and markers found in the file itself. The validation data is for checking a hypothesis, never for encoding an answer. A parser that special-cases the tables it has already seen is not a parser, and the corpus is full of vintages the next file will not resemble.
A handful of quirks account for most of the difficulty, and they are more interesting than the process of finding them.
The first is how the cells are laid out. A table nests its dimensions positionally, innermost to outermost, into fixed 256-byte pages. Exactly one dimension straddles a page boundary: as much of it as fits stays in the page, and the remainder becomes windows addressed through a directory. Which dimension that is depends entirely on the table’s shape — on modern census tables it’s geography, on a 1981 profile it’s a data dimension and geography sits inside the page instead. Nothing in the file announces which case it is. The effect is that tables from different eras look like they need entirely different decoders, when in fact there is one encoder and one set of rules applied to different table shapes.
Everything in that nesting is padded to powers of two. A dimension with five members will typically occupy eight slots, and one with ten members can occupy thirty-two. A format that goes to real lengths to avoid storing zeros is perfectly happy to leave two thirds of its addressing space empty, because power-of-two strides turn address arithmetic into bit shifts. Space is saved where it was expensive in 1990 — on disk — and spent freely where it was cheap.
Then there is byte order. Whether a given cell was written at all is recorded in a bitmap, and the bitmap’s bytes are stored in swapped pairs — byte 0 and byte 1 exchanged, 2 and 3, and so on — while the values that follow are stored in file order. It’s an artifact of whatever 16-bit machine the format was designed on around 1990. Read straight through, the bitmap almost lines up, which is the awkward part: it’s off by a bit here and a byte there, never consistently enough to look like a byte-order problem rather than a subtly wrong addressing scheme.
The same 16-bit heritage shows up in the file’s own pointers. The header stores the location of the page directory in two bytes, which stops working the moment a table grows past 64 kilobytes — and essentially all of them do. The high bits are simply gone. You recover the real location by adding multiples of 65,536 to the stored value until one of them lands on something that reads as a valid directory. The format outgrew its own address space and nobody widened the field.
The quirk with the most consequences is that the presence bitmap addresses members by slot rather than by position. Each dimension declares a fixed number of slots, and every slot is flagged as live, deleted, or never used — a deleted one still carries the member’s code, just without the length prefix a live one gets. So the file records not only the categories a table has but the ones somebody removed while building it: one Business Patterns industry axis carries 929 industries across 949 used slots, with 20 deleted. This is the clearest sign that IVT was never really an export format. It is a live, editable database file that happens to get shipped. It also has teeth, because a hole is invisible if you assume members are contiguous: read them positionally and every member after the gap is silently attached to the wrong data.
Last, the hierarchy among members is stored as whitespace. Census dimensions are deeply nested — “Total income” contains “With income” contains “Under $10,000” — and the format encodes that nesting as leading spaces in the display label. There is no parent field, no depth field. The tree exists only as indentation inside a string meant for a table header, which makes the structure exactly as durable as the presentation. And the indent width is not even constant across files: most tables use two spaces per level, but the census of agriculture tables use one, running Canada, province, agricultural region, census division and subdivision across zero to four spaces.
Nothing about the format is sophisticated. That is exactly what makes it hard — there is no elegant structure to reason your way toward, just a pile of small decisions someone made thirty-five years ago and never made public.
That is a tour rather than a specification. The byte-level account — page geometry, the header slot table, the codebook blocks, the bit-level slot records — is written up as the file format article, which also ships with the package as vignette("ivt-format", package = "canivt"). Underneath it sit the working notes the parser was built from and is still checked against: a byte-marker catalog, which the test suite verifies against the code so the documentation cannot drift from the recognizer, and a decode history recording how each vintage was cracked and why each invariant is worded the way it is. Both are in the repository and both ship inside the package, via system.file("notes", package = "canivt").
canivt
The canivt package provides the functionality to read and process IVT files and export the data and metadata. Metadata matters as much as the data here. The package recovers the dimension members and their hierarchy, the geographic names and standard identifiers (DGUIDs and their older equivalents), the data-quality flags, the footnotes — including which member each one is attached to — and the French labels alongside the English, all from the same single file. Those are precisely the things the Beyond 20/20 browser makes you extract one at a time, or not at all.
Additionally, it provides functionality to explore and search IVT data from two repositories, the Statistics Canada repository of standard release census data and the Borealis dataverse, a crowd-sourced repository of, among other things, IVT files from (custom) census data extracts. The latter are data tables individual researchers have made available to the general public. Either because sharing their custom extracts is the right thing to do, or because Tri-Council grants now require to make them publicly accessible and Broealis is a convenient platform to do so.
Ideally these custom tabulations on Borealis are unique extracts that are not available from the official StatCan repository, but unfortunately the Borealis repository is heavily polluted by researchers simply posting standard Statistics Canada products. This practice creates no additional value and pollutes the Borealis repository, making data discovery unnecessarily difficult. And it complicates data provenance, where people might cite the researchers who uploaded the data instead of referring directly to the standard StatCan catalogue number when citing the data source.
Additionally, there are many people who have pulled custom tabulations and never made them publicly available, if you have IVT files that could be of value to others, please consider adding them to the Borealis dataverse, whether they were funded via Tri-Council grants or not.
The canivt package is fairly mature at this point. Every release runs against a regression ledger of 133 standard and custom tables — 268 million cells, spanning the 1981 through 2021 censuses plus a long tail of survey and business-register products — and the ledger records the exact cell count of each, so a change that quietly shifts a single value fails the test suite. On six reference tables the decode is validated cell for cell against the published CSVs. Anything the parser cannot read with confidence is refused outright rather than guessed at, and every heuristic that does fire raises a warning naming itself, so a questionable value is never silently mixed in with good ones. The coverage tracker records what each table was validated against and to what residual, table by table.
That said, it may well still choke on some IVT files. If it does, please open an issue with a link to the offending IVT and a brief description of the failure.
Echoes of Beyond 20/20 IVT
An interesting by-product of this decoding exercise has been understanding the echoes of the format in modern release data. The New Dissemination Model (NDM) is the standard way Statistics Canada releases data today, and it is well thought out and broadly aligned with modern practices. But the census division was quite late to it, disseminating via the NDM only from the 2021 census, and then only after significantly degrading the NDM standards. Census data arrives in semi-wide form, and with a weird treatment of zero-value data when querying by table and coordinate. Neither has any place in a modern data product, but both are clearly echoes of the Beyond 20/20 IVT file encoding.
The zero handling is worth watching happen. Table 98-10-0040 crosses structural type of dwelling with household size, for Canada, the provinces and territories, and every census metropolitan area and agglomeration. It is published both as an IVT file and through the NDM, it is small enough to walk coordinate by coordinate in full, and — usefully — it contains all three of the things a cell can be when it is not a number: a zero, a suppressed value, and a not-applicable.
Nine hundred and thirty-six cells never made it into the file. Putting them back is the reader’s job, which is easy enough: the dimension members come back as factors carrying their full member list, so completing the grid is a one-liner. Here is one slice of it — the tall apartment buildings of Corner Brook, Newfoundland.
<code>
corner_brook <- tab |>collect_ivt() |>filter(geo_uid =="2021S0504015", # Corner Brook CAgrepl("five or more storeys", structural)) |>complete(household, fill =list(value =0))corner_brook |>select(household, value)
# A tibble: 8 × 2
household value
<fct> <dbl>
1 Total - Household size 5
2 1 person 0
3 2 persons 5
4 3 persons 0
5 4 persons 0
6 5 or more persons 0
7 Number of persons in private households 5
8 Average household size 0
A handful of households, all of them two-person — the counts are rounded to a multiple of five, which is why they all read 5 — and then, in the last row, an average household size of zero. Which is not a number anyone measured.
Now ask the NDM for the same eight cells. It addresses them by coordinate — the member numbers of each dimension, dot-joined and padded out to ten. Corner Brook is geography member 3 and that dwelling type is member 7, so the eight cells above are 3.7.1 through 3.7.8.
# A tibble: 8 × 5
coordinate status value status_code suppressed
<chr> <chr> <dbl> <int> <lgl>
1 3.7.1.0.0.0.0.0.0.0 SUCCESS 5 0 FALSE
2 3.7.2.0.0.0.0.0.0.0 FAILED NA NA NA
3 3.7.3.0.0.0.0.0.0.0 SUCCESS 5 0 FALSE
4 3.7.4.0.0.0.0.0.0.0 FAILED NA NA NA
5 3.7.5.0.0.0.0.0.0.0 FAILED NA NA NA
6 3.7.6.0.0.0.0.0.0.0 FAILED NA NA NA
7 3.7.7.0.0.0.0.0.0.0 SUCCESS 5 0 FALSE
8 3.7.8.0.0.0.0.0.0.0 SUCCESS NA 0 TRUE
Three of the eight come back as values. Four fail, and they fail in exactly the same way as a coordinate that was never in the table to begin with. Nothing in the response distinguishes “this value is zero” from “you asked for something that does not exist”, so a client walking a table by coordinate cannot tell a real zero from its own bug. Asking those coordinates for series metadata rather than data gives the same answer: an empty title and status code 2, either way.
The eighth is the interesting one. Average household size is not zero, and it is not absent because it is zero — it is suppressed, withheld for confidentiality, and the NDM says so plainly: the request succeeds, the value is null, and securityLevelCode marks it. The API has a full vocabulary for this. Eleven status codes run from .. (not available for this reference period) through ... (not applicable) to F (too unreliable to be published), alongside that separate confidentiality flag. Forty-nine cells elsewhere in this same table carry ... — an average household size for a dwelling type that nobody in that area lives in.
So the NDM distinguishes three different reasons a cell can be empty and reports two of them. The zero — the one case it could state outright at no risk whatsoever — is the one it drops. Status code 2 in that list even exists for precisely this territory: “value rounded to 0 where there is a meaningful distinction between true zero and the value that was rounded”.
The IVT distinguishes none of the three. It stores 11,592 cells, and the 936 it leaves out are 877 zeros, 49 not-applicables and 10 suppressed values, all written as the same nothing. That is why the completion above produced a false zero: there is nothing in the file to complete it any other way.
There is a third way to get the table, and it is the only one that tells you the whole story. Every 2021 census NDM table also has a bulk CSV download — what cansim::get_cansim() fetches — written out in full whatever the cube happens to store.
# A tibble: 8 × 3
`Household size (8)` VALUE Symbol
<fct> <dbl> <chr>
1 Total - Household size 5 <NA>
2 1 person 0 <NA>
3 2 persons 5 <NA>
4 3 persons 0 <NA>
5 4 persons 0 <NA>
6 5 or more persons 0 <NA>
7 Number of persons in private households 5 <NA>
8 Average household size NA x
There is the same slice, complete: the zeros spelled out as zeros, and the suppressed cell left blank with an x beside it. Tabulating the whole table by symbol recovers all four classes at once, and the cells the CSV leaves unmarked are, cell for cell, exactly the ones the IVT stores.
<code>
csv |>count(symbol =ifelse(is.na(Symbol), "<none>", Symbol),value =case_when(is.na(VALUE) ~"blank", VALUE ==0~"zero",TRUE~"non-zero"))
# A tibble: 4 × 3
symbol value n
<chr> <chr> <int>
1 ... zero 49
2 <none> non-zero 11592
3 <none> zero 877
4 x blank 10
Three ways of asking one table for one number, and three different accounts of what an empty cell is: the file leaves all of them out, the API reports two of the three, the CSV states all three. Only the last is unambiguous, and it is the one you cannot use when you want a single cell rather than the whole table. It has a quirk of its own, too — a not-applicable is written as 0 with a symbol beside it, so a reader who takes the VALUE column at face value sees 926 zeros where there are 877.
What makes the API behaviour genuinely awkward is that it is not consistent across the census cubes. Whether a coordinate query can answer with a zero is settled per table, by whether the zeros were written out when the cube was loaded, and the NDM’s own metadata gives that away: it reports the number of data points stored, which you can compare against the size of the full cross-tabulation.
Across all 525 census cubes, 69 write their zeros and 456 do not, and this split does not seem to depend on file size, complexity or date of the product.
What it does track is what kind of number the table holds, and that is visible in the IVT file itself — the last column above. Tables of counts, the rounded whole numbers that come out of a census tabulation, arrive sparse. Tables of rates, averages, percentages and quality indicators arrive complete. Across the forty-odd census tables we measured, the fraction of whole numbers in the IVT separates the two behaviours cleanly with nothing landing in between, and it held on the tables we picked out after forming the rule. Which makes sense, in a backwards sort of way. A count table inherits the convention that an absent cell means zero, because that is how the IVT wrote it; a table of averages has no such convention, since 0.0% is a computed number like any other, so every cell gets written out. The zeros do not go missing because anyone decided they should. They go missing when the table is the kind of table whose zeros were never stored in the first place. (There is a second, smaller group of complete cubes: tables whose IVT has almost no empty cells to begin with, which arrive whole for want of anything to drop.)
None of it looks arbitrary once you have seen the bytes. An IVT table stores only its non-zero cells. Each page carries a fixed-size bitmap saying which cells were written, and the values follow in bitmap order — so a zero is simply a bit that was never set, costing nothing. There is no per-cell status byte anywhere in the format, and no per-cell suppression flag; the only suppression the format can express is of a whole geography at once, stored as a page with no cells in it. Corner Brook’s suppressed average had nowhere to go but the same empty bit as the zeros beside it. That is a thoroughly sensible design for 1990, when the alternative was writing eight bytes of zero for every empty cell in a sparse cross-tabulation. It is a much stranger thing to inherit in a REST API three decades after the storage constraint that motivated it disappeared.
The semi-wide layout has the same lineage. An IVT file is a cross-tabulation: dimensions, members, and a value at every coordinate. Reshaping that into tidy long form is a step the Beyond 20/20 browser never had to take, and the census division perpetuated this by amending the NDM format to accommodate this. All other NDM tables are in tidy long-form.
Upshot
Being able to read IVT files programmatically has been on our list for a long time. Hopefully others will find this useful too. Older census data and custom tabulations provided by StatCan are available data, the canivt package turns them into open data by making the data and metadata readable by open statistical software. Statistics Canada data quality is generally very high, and the work that has gone into the New Dissemination Model shows that some effort has been paid to data governance. That makes it all the more astounding that Statistics Canada still has such large and lasting gaps in data governance in other parts of their data dissemination system. And quite possibly also in their non-public data, it’s hard to believe that these are just issues with external data, and the degradation of the NDM for census data releases point to much deeper underlying issues.
For not the package lives on GitHub only, we will spend some more time testing and collecting feedback before submitting to CRAN.
Reproducibility receipt
<code>
## datetimeSys.time()
[1] "2026-07-26 23:20:47 PDT"
<code>
## repositorygit2r::repository()
Local: main /Users/jens/R/mountain_doodles
Remote: main @ origin (https://github.com/mountainMath/mountain_doodles.git)
Head: [f018316] 2026-07-27: condense post a bit