Context

In a first anywhere in the world, a court in the Netherlands recently stopped a digital identification scheme for reasons of exclusion. This has a context for similar artificial intelligence systems worldwide, especially at a time when identity, citizenship and privacy are pertinent questions in India.

What was the scheme?

• SyRI (System Risk Indicator) is an identification mechanism.
• The Dutch Ministry of Social Affairs developed SyRI in 2014 to weed out those who are most likely to commit fraud and receive government benefits.
• Legislation passed by the Dutch Parliament allowed government agencies to share 17 categories of data about welfare recipients such as taxes, land registries, employment records, and vehicle registrations with a private company.
• The company, called “The Intelligence Agency”, used an algorithm to analyse data for four cities and calculate risk scores.
• The selective rollout was conducted in low-income and immigrant neighbourhoods, which have a higher number of beneficiaries.
• Elevated risk scores were sent to relevant government arms, which stores these on government databases for a maximum of two years.
• The government, in that time period, could open an investigation on the targeted person.

The issue:

• Recently, a Dutch district court ruled against SyRI (System Risk Indicator), because of data privacy and human rights concerns.
• While the Hague district court found using new technology to control fraud was acceptable, it held SyRI was too invasive and violative of the privacy guarantees given by European Human Rights Law as well as the EU’s General Data Protection Regulation.
• Legal criticism alleged that the algorithm would begin associating poverty and immigrant statuses with fraud risk.
• The court found that opaque algorithmic decision-making puts citizens at a disadvantage to challenge the resulting risk scores. The Netherlands continuously ranks high on democracy indices.
• The court ruled that SyRI was violative of principles of transparency and data minimisation laid out in their General Data Protection Regulation.
• Other European tech initiatives have been stalled by the regulation, including a facial recognition system on students in Sweden and France.

How relevant is this for India?

• Similar to the Supreme Court’s Aadhaarjudgment setting limits on the ID’s usage, the Hague Court attempted to balance social interest with personal privacy.
• However, the Aadhaar judgment was not regarding algorithmic decision-making; it was about data collection.
• The ruling is also an example of how a data protection regulation can be used against government surveillance.
• India’s pending data protection regulation, being analysed by a Joint Select Committee in Parliament, would give broad exemptions to government data processing in its current form.
  • Similar regulations by the US:
• India’s proposed regulation is similar to the US in the loopholes that could be potentially exploited.
• Hence, attempts to ban facial recognition in cities such as San Francisco have not had the same success as attempts in Europe.
• A system somewhat paralleling the Dutch SyRI system was a risk-scoring software being used by US court systems to establish bail times.
• The US Supreme Court declined to hear a related case in 2017.

Global countries taking note of the ‘ruling’:

• Digital ID systems are being rolled out at a fast pace in places like Kenya, Philippines, Nigeria, Mexico, and more.
• Experts worldwide have been watching the Netherlands case throughout, and agree that the ruling will ripple beyond south Rotterdam.

The ruling sets a strong legal precedent for other nations to follow. This is one of the first times a court has stopped the use of digital technologies and abundant digital information on human rights grounds.