How to Build a Secure, Auditable Voting System

By David B. Black

I am a computer and software guy with experience in building systems, networks and security in multiple industries. I know only what I’ve read about the voting systems in place in the US. The basic information that’s widely available is enough to make it clear that today’s voting systems were designed using circa 1990 principles without regard to the security methods that were used by the best systems even at that time. In the light of modern systems and security design they might as well have large, blinking neon signs on them reading “Cheat Me.” This has NOTHING to do with the 2020 election, just as building a secure bank vault has nothing to do with whose money it holds. We ALL care about having safe, secure and accurate voting.

A Modern Voting System

It’s possible to build a voting system that is transparent, fully auditable and extremely difficult to cheat. Each vote would not just be recorded but also logged simultaneously in multiple places, locally and in the cloud and available for public and private review within seconds. This by itself would eliminate huge amounts of potential fraud by getting humans and local physical devices out of the counting loop and above all by ending the secrecy of today’s proprietary closed systems.

Some elements of a secure voting system vary for in-person voting and mail-in voting but ballot counting is the same:

Custom voting equipment of any kind should be eliminated and replaced with COTS (Commercial Off-The-Shelf) equipment.
The only local operations that should be performed are voter identification and vote capture

Paper ballots (and envelopes, if relevant) should be scanned and the encrypted images sent securely, in real time, to multiple locations in the Cloud.
Assure that each paper has a unique ID and include full log information, things like date/time, source IP, etc.

All processing possible should be done in the cloud for both in-person and mail voting:

in multiple places and clouds, with results compared
fully logged real time in multiple places

Converting images to votes should be done distributed in parallel

The knowledge of whether a vote was made and who it was for should be kept separate

The cloud tallied vote made by a person should be shown to them seconds after they vote in person
Tallies are updated in real time by voting location including rejections with reason codes, all public
While some on-site software is unavoidable, it should be minimal and become open source
All places where voting takes place or ballots are counted should be under total video surveillance, streamed to multiple places, using COTS equipment
Each jurisdiction enters the voting data uniquely under their control. For example, a state would enter information about the candidates who qualified for the ballot for governor, the US senators from their state, etc. The same would be done at the County and local levels, for example a town would add town council candidates, school board candidates, etc. In each case they would enter all the information that would appear on an electronic or paper ballot and more.

Each jurisdiction would enter additional voting requirements such as what kind of ID is required, whether signatures are required and/or checked for mail ballots, etc. All such parameters are public and changes are logged.
The parameters made public actually control the software operation in addition to being publicly readable documents, leaving no room for secretly interfering with the operation of the software.

Scanning and Processing a Ballot Overview

The process of scanning and processing a ballot would be nearly identical whether voting in person or by mail. This by itself is a big step ahead in efficiency and security. This means that a person fills out the same ballot in the same way whether at an in-person center or voting by mail. They see how they voted by looking at where they marked the paper. The only thing done locally by electronics during either in-person voting or mailed ballot processing is scanning the ballot (like taking a picture of it) and sending the unprocessed, encrypted image to multiple secure storage locations in the cloud, along with logging this in the cloud. Once ballot images are sent, stored and logged they can’t be altered or discarded. This reduces to a bare minimum the exposure to error or fraud at the processing location.

The fact that images are converted into votes and tallied within seconds by distributed software working in multiple clouds comparing their results and then showing them to the voter further reduces fraud – each voter can see whether their vote has been tallied correctly.

Various jurisdictions have methods to prevent someone voting more than once today. This is a big subject. Briefly, the key is assigning each qualified voter an ID. The vast majority of places already do this in some way, which is how they can mail ballots to registered voters and make sure no one votes in duplicate. A uniform system of voter ID’s would be created, each linked to whatever local method is in use to minimize disruption. The ID’s would be printed on all ballots mailed and affixed to ballots made available to people voting in-person. The ID’s would be scanned as part of the ballot and processed by software in the cloud immediately. Because this is done within seconds of ballot presentation, attempts at duplicate voting in any way would be caught and prevented from being included in vote totals.

Ballot Processing Center Setup

Most of a ballot processing center would be the same for in-person voting or for processing mail-in votes. In fact, centers could be used for both purposes, even at the same time.

Each center would have a unique name displayed near its entrance. There would an appropriate number of workstations, perhaps tables with chairs, for processing ballots. Each workstation would have a unique name prominently displayed on a sign, for example the name of the location and the number of the workstation. Each workstation would be equipped with a modern high resolution color scanner, preferably duplex (scans front and back in one pass). If for any reason duplex scanning is not practical or available, the operator would scan in two steps, front and back. But duplex is preferable because it eliminates a step and reduces the chance of operator error. There are a variety of COTS scanners with the quality and resolution to handle the job. The scanner would be connected to a simple off-the-shelf laptop computer with a tiny amount of custom software whose only job would be to control the scanner and copy the scanned images to multiple locations in the cloud, with each step logged multiple times. The laptop’s camera would be live and also streamed to the cloud.

In order to minimize the possibility of hacking, the computers used would be recent, bare, off-the-shelf models. Since all the leading laptop operating systems are huge, incredibly complicated bodies of software that have security holes and need regular patches, the software used on it would a small amount of custom compiled C code that would be made open source to enable the software community to find and correct errors and security issues. During room setup the physical machine would be connected to the internet; the custom control code would be downloaded, wipe the machine clean and make itself the only software on the machine. An operator would point the machine’s camera to the ID sign on the workstation, which the software would read, register itself to the cloud and display on its screen. It would then communicate with the cloud control software that would cause other tests to be made. There’s lots of detail here that is beyond the scope of this brief description, but the point is that the opportunities for local shenanigans or errors are brought to near zero.

Each room used for handling ballots would be equipped with multiple modern security video cameras streaming to the cloud to assure that nothing improper was done during ballot processing. Old-style proprietary video security systems would not be used. The video stream would be made available at least to appointed observers and perhaps also to the public.

Voting in Person

A voter checks in much like today, presenting ID as required. If they are deemed qualified to vote they are given a paper ballot with their unique voter ID affixed to it. They go to a private desk area to fill out the ballot. They take the ballot to an available scanning workstation and feed their ballot to the machine. After the ballot is read it is put in a discard bin.

If the voter chooses they can pause at the workstation for a couple seconds and wait for the results of processing their vote to appear on the screen of the laptop at the workstation. When the image of their ballot has been securely stored in the cloud and their votes extracted from the image, validated and tallied, the candidates and issues they voted for are displayed on the screen. The voter knows that not only have they voted but that their votes have been recorded and correctly tallied without delay, and with no further steps required. They can then leave the workstation, pick up their “I Voted” sticker if they like and leave the place of voting.

Additional security would be provided by asking each voter to not just look at the things they voted for displayed on the screen as recorded and tallied, but also to validate that the votes recorded for them do indeed match the votes they think they made. This would be done by giving the user two screen choices: “votes are correct” and “votes aren’t correct.”

Exactly how re-voting is handled must be done carefully, because it’s an opportunity to introduce fraud. One way to handle it would be for the user to touch the votes on the screen they want changed and once the voter is satisfied to click to submit the corrected vote. I suggest a paper-based method be used to assure that this potential door for hacking to enter remains closed. For example, the voter would take one of a stack of blank “vote correction” sheets, sign it, copy a number onto it that is displayed on the screen and scan the sheet to prove their intention to make a correction. The frequency and distribution of corrections should be monitored closely in real-time to detect bad things happening.

Voting by Mail

Mail-in ballots that arrive by any method would be immediately processed in the same kind of room as used for in-person voting – it could be the same room! There is no reason why the two kinds of voting couldn’t be done at the same time.

The mail-in ballot processing operator would:

pick up the top envelope from the stack of unprocessed ballots, the in-box
show the envelope to the camera being sure to avoid obscuring the unique code on the envelope

This image provides a double-check that the ballot was processed

scan the envelope (duplex, front and back, in one step)
remove the contents of the envelope and place the envelope into a processed box, the out-box
if the contents is a secrecy envelope, scan it (duplex, front and back), remove the ballot and place the envelope in the out-box
unfold the ballot if required and scan it
place the scanned ballot in the out-box.
Repeat until the in-box is empty.

Each log entry would contain a date/time stamp, location GPS and name and workstation ID, links to the corresponding images that were scanned and to the place in the video log that captured the scanning process. There would be software running in multiple cloud locations that would process each log entry as it was written and make counts and summaries publicly available via API and web pages. The same software would produce real-time roll-ups so that anyone could follow the progress of ballot registration.

Many states have systems to enable mail-in voters to see whether and when their ballots have been received and then whether they’ve been accepted. Each ballot was printed with a code uniquely assigned to a voter. As soon as the log entry was written for the images the fact that the ballot with the voter’s ID was received would be sent by the cloud software to the state system for updating that voter’s ballot status to “received.” After complete processing, which would normally just be seconds later, the status would be updated as appropriate to “accepted” or “rejected.” The system would provide the state the reason the ballot was rejected, which could include duplicate ballot, lack of signature, ballot not in the right envelope (ID mis-match), etc.

Turning the Ballot Images into Votes and Totaling Them

Ballot images are captured and stored securely in the cloud along with detailed logs in the same way for in-person and mail-in ballots. The only differences are that additional images are stored for mail-ins – the envelopes, both outer and security.

The crucial process of “reading” the images would be performed by software running in the cloud. Multiple copies would be run on different servers controlled by different entities. The output of each major result from the software would be sent by that software to its siblings, all of whom would have to declare that the results matched in order to proceed. They would then “report” their results to multiple queues and log files. This method of operating copies in parallel and comparing results is an established method of assuring against malicious actors and assuring fault tolerance.

In addition, the software would not be in a single block. The images would be broken up and the different parts processed by different pieces of software. This is a modern approach to building distributed software that is usually called micro-services. It is used to build highly scalable, secure and fault-tolerant systems out of components (called “services”), each of which has a small, isolated job to do. Using this method, the software service that decides whether a signature is present or whether a box or circle has been filled in to indicate a vote has no way of knowing who or what the vote is for, and therefore no way to slant the results. In the unlikely event of a successful hack of one or two pieces of software, it wouldn’t be able to hack the results.

To process the images the software uses modern OCR (Optical Character Recognition) algorithms. OCR is used on both the preprinted text and whatever the voter entered on the ballot. OCR is a mature technology, with software libraries widely available and deployed in production today. OCR is used by bank apps to enable customers to electronically deposit checks using just an image of the check taken by smartphone. Higher quality versions are in broad use today that enable people to scan and submit bank statements, pay stubs, ID’s, and many other document types in order to apply for a loan on-line, become an Uber driver and a host of other purposes. OCR software is no less accurate than human readers who type what they read, and arguably more accurate.

It’s important that the image processing be done in small steps so that no one piece of software processes a whole ballot. This serves multiple purposes including keeping a voter’s votes secret and maximizing the defense against hacking. Once an image was scanned, there are proven techniques for accomplishing this that are faster, more accurate and secure than the processing that is done today for both in-person and mail-in ballots by humans and local machines. Here are the highlights:

Every image is stored with strong encryption in deep cloud storage with multiple backups.
Paper ballots today do not normally have the voter’s name on them. The name appears only on the containing envelope. This is a good practice and should be maintained.
The image of the ballot itself would be processed by program that would use OCR to “read” all the printed text much like a person would.
The OCR would pick out each candidate name and issue description and identify the area on the image in which a voter is supposed to fill in a circle in order to vote.
The same program would create a unique ID for each snippet of the ballot image that the voter could have filled in and write each little image to a new file along with the identifying information, put it on a queue and create a log entry.
Multiple copies of separate “vote recognition” programs would be constantly reading the queues and reading the vote snippets. They would evaluate each snippet for whether it had been filled in or not according to uniform standards – without having any information about where the vote was made, which candidate the image was associated with or who the voter was. Each program would then write its results to a queue and log file itself. This file would contain the vote recognition program’s unique ID, the unique ID of the snippet and its judgment of whether or not it had been filled in.
Separate “vote collector” programs would read the queues of the “vote recognition” programs to gather all the votes in a single ballot together. These would be written to a queue and log of their own.
The first ballot-reading program would read the collected vote queue, use its data to see which vote was for which candidate as read by it from the ballot and write the final vote tally into a multi-copy log file. The most important data in each log entry is the list of candidates who received a vote. The unique ID of the image would also be in the log entry, linking them to make a completely transparent audit.

It is essential that this step be performed in parallel by multiple copies of the software running in completely separate clouds and the results only released when all the copies reach consensus.
If there are enough software copies, say a dozen, then if all but one report the same results, the exception is logged and discarded.
If something goes wrong with a cloud location or service, so long as most of the services copies are alive the process would be unimpeded.

Finally, vote tally programs would read the vote logs in real time and update vote totals in real time for anyone to see.

Each individual in-person vote would in addition be immediately returned to the voting site and specific workstation for display to the person who voted.

The steps described above provide the general idea and specifics for ballots. Some fraction of the votes will be mail-in. They will be marked as being mail-in during the scanning process and will consist of more images; for a typical vote there would be six images, two for each of two envelopes and one ballot. Depending on the requirements set the system would:

OCR and check the envelope.

Record the ID on it and assure that the person hasn’t already voted.

OCR and check the inner envelope.

Apply the same ID check and assuring that it matches the containing envelope.
Apply any signature requirements (see below)

Process the ballot as already described, checking the ID for match.

Using modern technology the entire process just described for either in-person or mail-in should take place in seconds.

Suppose 200 million people voted and all the votes arrived in a 10 hour period, which they wouldn’t. This is 555 votes per second. Suppose that just a hundred machines were used; many times that are available in each of the major cloud services. This would mean that each of the 100 machines would need to handle roughly 5 votes per second. Even with all the parallel and additional processing and checking, this is a laughably trivial load for modern machines and networks to handle. The system would be self-managing like any good micro-service mesh and automatically scale up the servers if and when needed.

This is not a software specification. I’ve written many such documents and am well aware that there are a number of conditions and details I have not addressed. This is an overview to give a sense of the overall approach.

Checking Signatures

I have purposely left the issue of signatures to the end. I don’t want to address the question of to what extent they should be required and checked.

These are the main elements of a solution that can be applied to whatever extent is decided. First seeing whether there is a signature:

The program that handles the image (probably not the ballot) that can have a signature would perform OCR on the image to identify and extract the portion of the overall image that could contain a signature, much like the ballot processing program extracts the areas on the ballot that the voter could have filled in.
Much like the process described for seeing if a voting circle has been filled in, a separate program receives the signature image and decides whether there is a signature. The several such programs that look at this image assure that they concur and the results are logged and sent back.
This information is then read by an additional signed-vote program. It takes the input from the signature-page-reading program and the is-there-a-signature program, and combines it with the input from the ballot reading program, creating the log that the vote tally programs read. This enables them to create separate talles of valid and invalid votes.

If signature matching is also required, additional steps must be performed. In short they are:

The voter rolls with signatures should be scanned in advance of voting.

The same physical equipment should be used as for mail-in ballot processing
The software should get the name and address of the voter, the ID of the voter as used by the relevant authority and an image of the signature.
Unfortunately, the exact method of doing this may vary by jurisdiction. I don’t know enough about current practice to handle this issue with confidence.

When ballots are mailed to voters, the ID’s placed on the mailed documents should be put into a secure online table to enable the signature images as scanned during registration to be matched with signatures made on voting documents.
During vote counting the same process to extract signature images as described about should be followed. The process to determine where a non-blank signature exists should also be followed.
If the signature doesn’t exist at all, the vote is invalid and should be handled as above.
If the signature exists, there are two ways to handle it, which could be done in any combination.

The automated method would be done entirely by software. Probably using a machine language method called convolutional deep learning, neural networks would be trained with huge samples of real-life signatures with multiple signatures from the same person. For example check archives could be used for this purpose.
The widely used human-in-the-loop method would show workers pairs of signatures on the same screen with no other information. One would be the original signature and the other would be the signature on the mail-in ballot. The worker would enter one key for “match” and another key for “no match.” No other information would be provided. The system would assure that the humans who saw the signatures lived far away from the signers, but in any case the checkers would only see each pair of images for 5 seconds or less.
Each pair of signatures could be presented to multiple human and/or automated checkers and the results compared.
This is a huge subject, but elaborations of these basic procedures would produce results that were no worse than today’s physical methods, with very low probability of bias entering into the process.

The methods I’ve described here can be applied to other things the voter may be required to write on an envelope, including for example a date.

Software Controls and Parameters

All software has controls and parameters to adapt to local requirements and conditions. In primitive systems like today’s proprietary machines, each machine is set up by a local systems administrator, who can set and change the machine’s parameters and controls at any time.

In this system, all controls and parameters for all software are contained in a centralized system of tightly controlled and logged editable metadata to control all operations of the system instead of typical administrative control and parameters. This is a key aspect of making a diverse set of micro-services fully controlled and coordinated, while conforming to the requirements and conditions of each jurisdiction. The metadata would be organized in a hierarchy with inheritance, so that rules set at a state level would automatically inherit down and control the relevant aspect of domains within its jurisdiction. The hierarchy would establish a parent/child relationship between blocks of metadata so that counters such as voter and candidate vote totals would automatically roll up. There could be multiple hierarchies enabling for example a voting location to belong to just one town, but the town to belong separately to a county and a congressional district.

The metadata would control exactly which images constituted a mail-in vote, the tests to be applied for validity, the reason codes used for rejection, etc. This is an important aspect of making the system operation fully transparent – the metadata could be used to generate a human-readable document on the web anyone could read.

The controls for creating and editing the metadata are crucial. There would be a CRUD (Create Read Update Delete) matrix between each permission group and each metadata block instance, for example a state. A person who belonged to the permission group for a particular state with C permission would be able to enter and edit candidates and issues to vote for. Since this is done only once per election and the data is so small, it’s likely that such high-level permissions would be restricted to a couple of centralized people with security similar to that for launching attack rockets. Local security would be for creating voting locations and stations. Things like whether signatures are required would be made at the appropriate controlling jurisdiction level. In any case all changes would be made in collaboration with a central group including verbal interaction with multiple people to prevent hacking of any kind.

In all cases setting and changing parameters is highly infrequent but dangerous, which is why gaining access is made burdensome and the results fully public. Changes would be halted at an agreed time prior to an election and before early voting if any.

Because all control parameters and their settings are handled in this way with public viewing of the settings, there is no need to do any software administration and update for any reason, which makes it possible for the software source code itself to be made available for public inspection.

Building the System

The system described here can be built quickly and inexpensively if the appropriate skills and entrepreneurial methods are used. Disaster, delays and ballooning expense would result from a typical corporate/governmental RFP process with endless meetings, reviews and input from “experts.”

Separate teams of a couple people each with appropriate skills could write each of the components/services described here. The toughest skills to get are the currently rare knowledge of bare-machine programming; therefore a preliminary step of running the software on a standard operating system could be used to get a working prototype. There are a few infrastructure things that would need to be agreed to, for example the exact methods for making calls among the mesh of services and otherwise coordinating their activities. It would be best if common tools like redis were used for reliable, super-fast queuing were agreed to and used when appropriate. The metadata control system would need to be built by a single team, but there would not be a much code involved. Its API would be accessed by all software services, probably just once at the start of operation.

The system could first be deployed in small scale with purely local elections for things like home-school associations. Cooperating government entities could make boxes of ballots from past elections available to try out the software.

Conclusion

One of the key benefits of the modern method of voting I’ve described is that it eliminates nearly all of the human setup and administration that is currently performed by thousands of people at vote-processing locations. It also eliminates the many thousands of error-prone human steps that are required to process votes, including things like USB drives that are currently moved by administrators from voting machines to web-connected laptop computers.

While there are lots of details I haven’t filled in, nothing I’ve described here should be foreign to software people on the front lines. Systems much like it are in production at scale in multiple industries. The wide-scale logging, parallel processing and comparison of results are standard methods for assuring that a fault of any kind, malicious or just random, doesn’t cause problems. While everyone, including me, has concerns about hacking, it’s well-known that the worst hacks are typically inside jobs, and taking people out of most of the processing goes a long way to increasing security. The chances of success would be greatly increased by making the software be a kind of open source so that anyone can point to vulnerabilities. For example, open-source Linux software runs something like 90% of the world’s web servers; it’s the gold standard for open and auditable while also being more secure than anything produced by a closed software group.

If a system like this were in use, everyone would be able to be confident that insiders of any variety weren’t using their power over the process to skew the results; except for the identity of the people voting, every step of every vote would be open to inspection by anyone in near-real-time.

Everyone should be able to support bringing voting up to modern standards.

The Patriot Post® · How to Build a Secure, Auditable Voting System