hermosawave, Getty Images/iStockphoto

Multiple organizations have approached President Joe Biden this week about a widespread overhaul of the country’s voting systems and technology due to the fact that dozens of states are using technology that is decades old.

Some experts are pointing to the “Voting Solutions for All People” (VSAP) initiative in Los Angeles County as a prime example of the kind of innovation needed to bring American voting up to date.

Drew Jaehnig, who spent more than 20 years with the US Defense Department before becoming a director at software company Bizagi, said 38 states are still using voting systems that are no longer supported by their manufacturer, putting their states’ voting mechanisms at risk. An additional seven states are exclusively using obsolete machines.

When most Americans think about voting, many imagine pens, paper ballots, and old-style hand-cranks. But Los Angeles County, the most populous county in the country, rolled out an innovative new voting system during the contentious 2020 election season that sought to bring US voting into the 21st century with the help of new tablets and other digital tools.

SEE: Big data’s role in COVID-19 (free PDF) (TechRepublic)

Aman Bhullar, CIO of the LA County Registrar, said in an interview that LA County is one of the most complex election jurisdictions in the country with 500 political districts and 4.3 million registered voters. Even before the COVID-19 pandemic changed almost every aspect of life, the county was testing out its new initiative, which Bhullar said sought to place voter experience at the center of its design.

Bhullar was quick to note that this is not “online voting” but instead the voting at polling locations with ballot-marking devices and electronic poll books that come with multiple accessibility options, multiple languages, and of course, top-end security features to make sure everything was safe.

“Like voting technology across the country, we were dealing with legacy systems. The systems were antiquated and put in place many, many years decades ago. Those systems worked for us back in the day but as technology has changed, with new, evolving security threats and an evolving voter population, we thought, ‘Let’s make this voting experience seamless for the LA County voter,'” Bhullar said.

“The results were a very successful and smooth run in November. Everything came together in perfect harmony.”

Bhullar explained that LA County began mulling changes to its antiquated voting systems nearly 10 years ago but struggled to find vendors who could handle the size, scale, and security needed to handle a population that large.

After working for years to find the right vendor, county officials decided the best path forward would be to create a custom-made system using tools from a variety of companies. LA County partnered with a company called IDEO, which came up with three concepts that were later refined into one model. The final design was an air-gapped portable ballot box with a touch display covering 13 languages with a QR Code reader and printer.

Voters are given a blank ballot that is inserted into the machine to start the process and the voter then makes selections through the touch screen process, which can be audio and Braille-assisted if need be. Once comfortable that the selection is correct, VSAP prints the filled-out ballot, and voters can then review the paper ballot for errors.

In the end, voters submit the paper ballot back into VSAP to cast a vote electronically and via paper.

Keeping track of devices

After years of working on both the hardware and software needed, Bhullar said he and others in the county began to think about the best way to operationalize the system. But part of the process was being able to track the 80,000 devices that would be used across the county.

Sam Gilliland, CEO of Cherwell Software, said his team began to work with LA County on creating a digital-based chain of custody system to monitor the devices and keep track of any maintenance done on them.

“They had warehouses full of devices and would ship them out to voting locations, so they have to track and ensure that those are getting to the right place. Once they get to the right place, they have to make sure they’re operational and that they are maintaining them,” Gilliland said.

Gilliland said security was a paramount concern of the county, so every device had a secure zip-tie lock on every device as well as a barcode seal to ensure that any time the device is used, someone is aware. If there is ever a problem with a device, the zip tie lock has to be removed, and that removal is recorded along with any fixes done on the device.

Bhullar noted that he can personally pinpoint when a device was opened, who opened it and who authorized the maintenance. They intentionally designed the entire system to be auditable in an effort to be as transparent as possible with the new system.

They can be certain that there was never any unauthorized access or even touching of devices because of the tracking system the county built specifically for these devices.

There were multiple devices rolled out within the VSAP system. The main voter experience was done primarily through the ballot marking devices, which had touch-sensitive screens with multiple options in terms of accessibility and language. Bhullar said the county worked with accessibility groups to help create a system that could work for everyone, regardless of disability.

Many groups had input on what would help the county design an equitable system as they sought to reimagine the voter experience.

“California is a paper-ballot based state. So to meet regulations and compliance, we had to have the paper ballot. We understand the importance of paper ballots and the need for security but at the same time, we wanted to give our digital experience to the voter,” Bhullar said. “But eventually what gets counted is still the paper. That is your official ballot.”

The county also rolled out electronic poll books, which are tablets that are used by poll workers to check in voters. These devices are also used to communicate back to the county voter registration database.

Although the voting is done on devices, the ballots are eventually printed out onto paper. The ballot-marking devices are not connected to the outside world at all, so people can be sure that their ballots are never tampered with, according to Bhullar.

The electronic poll books are connected to the internet and require their own sophisticated security program. Gilliland said Cherwell Software helped the county create an incident tracking system that immediately notified Bhullar’s team if a device was not working or had problems.

Gilliland noted that over the 11 days of voting that took place in November, almost 390,000 chain-of-custody records were created tracking every single device as they were used throughout the election season.

Bhullar added that multiple jurisdictions have seen the VSAP model and are interested in deploying their own version of it. LA County, he said, is lucky enough to have the means to create its own special system but that they eventually hope to make their source code open source so other counties can use it.

“We are sharing that knowledge with other jurisdictions where they can learn from us and eventually down the line, it’s a matter of time until other jurisdictions are able to take our source code and run with the system,” Bhullar said.

They tested the VSAP system multiple times before rolling it out for the elections in November. Ahead of the presidential primary in March 2020, they had to work out the kinks, including where the devices were warehoused, how they were transported to polling locations, and more.

In addition to the trial runs, they went to great lengths to keep the public informed about how the system would work and kept stakeholders in the county up to date on all of the changes being made.

They had a media campaign and even held open houses, where people could come in and test the devices for themselves.

“We let people come in, touch and feel the devices. Transparency was a big thing in my mind because I wanted to make sure that I was fully transparent. I welcomed oversight. I welcomed third-party testing,” Bhullar said.

“We started doing penetration testing by independent third parties in addition to what we were obliged to do as part of certification. We went to DHS, for example, and asked them to do penetration testing. When it comes to where the devices are loaded, we purposely made an area where observers, the public, can come in and watch us as we are preparing our devices.”

The area where they print and count the ballots is an air-gapped environment but has multiple observer areas where people can watch the count. The county also live-streamed the count on its website.

“We’ve shown to the world that this model can work. There are definitely lessons learned and it’s out there in public. In a year or so, we’ll be at a point where our code will be shared so that people can actually make use of at least the technology pieces or processes,” Bhullar said.

Security experts share thoughts and concerns

Some cybersecurity experts had minor reservations about the system. Brandon Hoffman, CISO at Netenrich, said VSAP was a positive effort but said there were still some outstanding questions from a cybersecurity standpoint that are not addressed in the documentation released online.

Hoffman said much of the information available was generic and that the county needed to provide specific details about how the ePollbook and ISB technology, along with counting machines mentioned, would be secured against malware and voter fraud.

Jaehnig said the VSAP initiative is interesting because it’s the very first of its kind.

“Functionally, the design worked well for this year’s election cycle and has broad applications to other markets, thanks to its bi-partisan and private-public design. It meets all parties’ concerns and has accelerated the voting process,” Jaehnig said.

“Additionally, the initiative may be a means to replace current systems in the 38 states using systems no longer supported by their manufacturers. Practically, the chain of custody of the machines for deployment, the 11 days of voting and post-election retrieval, and audit needed to remain rock solid, which required another set of asset management software and process to manage the machines in transit and while deployed.

The VSAP system also allows for easy upgrades in terms of security and technology, allowing states to keep everything current.

“Generally, VSAP demonstrated that more technology enables more voter voices to be heard, making voting more accessible for differently-abled and ESL voters. If you combine this with a robust audit and security capabilities, systems like VSAP show great promise,” Jaehnig said.

“However, we must exercise that promise with great vigilance to ensure the US voting system at large remains one of the most secure in the world.”