SureChEMBL 2.0 is here

SureChEMBL 2.0 marks a major evolution of the SureChEMBL system. Several core components of the pipeline have been replaced from annotation, to compound registration, to data provision. This new system will also make it easier to develop and deliver new functionalities within SureChEMBL. 

New data source

As part of SureChEMBL 2.0 the team has added the China National Intellectual Property Administration (CNIPA) to its list of patent authorities. SureChEMBL now integrates data from five authorities:

1. United States patent and trademark office (USPTO)
2. World intellectual property organisation (WIPO)
3. European patent office (EPO)
4. Japan patent office (JPO)
5. China National Intellectual Property Administration (CNIPA)

The data SureChEMBL obtains from each of these authorities varies. SureChEMBL receives rich content including attachments and full text for USPTO, EPO, and WIPO. However they will only receive titles and abstracts from JPO, and translated full text (no attachments) from CNIPA. See table below for full details.

Authorities	Kind	Language	From	Full text	Attachments
CNIPA	Applications	EN	1985	Yes (English translation)	No
	Granted
EPO	Applications	DE, EN, FR	1978	Yes	Yes
	Granted		1980
JPO	Applications	EN	1976	Yes(abstract)	No
USPTO	Applications	EN	2001	Yes	Yes
	Granted		1920-1949	Yes (abstract)	Yes
			1950-1975	Yes (abstract & claims)	Yes
			1976	Yes	Yes
WIPO	Applications	EN, FR	1978	Yes	Yes

The numbers

Here’s a snapshot of how many documents and chemical structures SureChEMBL has identified from each authority. While many patents are unrelated to chemistry (and won’t contain chemical structures), all documents remain accessible via the SureChEMBL web interface, keeping it a universal patent search portal.

Notably:

• USPTO (21M) has fewer documents than CNIPA or JPO, but contributes the most unique compounds (20M), thanks to the availability of the MOL and CDX attachments included in the Complex Work Unit (CWU).
  
• CNIPA (51M) and JPO (30M) lead in patent volume, but limited data access results in fewer compounds (4M and 193K respectively).
  
• EPO (9M) and WIPO (6M) have fewer patents but more chemical structures (13M and 11M, respectively), thanks to our image-to-structure extraction pipeline.

Authorities	#Patents	#Annotated patents	#Unique compounds
CNIPA	51,134,665	19,843,662	3,882,257
EPO	8,685,902	5,893,457	12,637,484
JPO	29,661,823	3,430,164	193,340
USPTO	21,243,929	11,261,705	19,903,544
WIPO	5,873,483	3,870,512	11,144,795

Introducing: SureChEMBL bulk data download

In response to evolving operational requirements, SureChEMBL has introduced a unified download system that merges the simplicity of MAP files and compound dumps with a new bulk data download system. For more information on the reasons for this change, take a look at their recent blog post. 

The bulk data is a new collection of a few core files representing the entire annotated SureChEMBL database. It is conceptually similar to the MAP files, but with much richer content. Here’s the schema they are using:

If you’re familiar with relational databases, the structure should be straightforward. One file contains the compounds, another the patent documents, and a third the compound–patent relationships. A fourth smaller file holds metadata about patent sections (e.g., title, abstract, description, claims, images, MOL attachments).

This format mirrors SureChEMBL’s internal relational database schema, meaning no duplication of compound or patent information. For instance, in MAP files, including the patent title meant repeating it for every compound–patent relationship, inflating file sizes. With the new schema, such repetition is avoided.

This structure also allows SureChEMBL to easily extend the data model in the future. Also, because each release is a snapshot of SureChEMBL at a given time, it’s self-contained and changes in future releases won’t affect previous ones.

Release frequency and format

Creating a full snapshot of SureChEMBL takes time, so for now, the team plans to deliver updates every two weeks. This is more frequent than the quarterly MAP releases, though not as fast as the former data client.

They are also transitioning to a new file format: Parquet.

Parquet is a columnar storage format widely used in big data platforms like Apache Spark and Hadoop. It’s designed for efficient querying, particularly for analytics that access only a subset of columns. Benefits include faster read performance, better compression, and support for complex nested data.

While interacting with Parquet files requires using a library (e.g., Pandas, PyArrow, Polars, DuckDB in Python), it’s very similar to querying a SQL database.

Here is how to get all the compounds for a given patent:

Converting Parquet to CSV

For those not comfortable with Parquet, conversion is simple:

If you’d prefer to use traditional formats like CSV, you can easily convert the files using the DuckDB CLI.

First install DuckDB (available for Linux, macOS and Windows).
Then, run the following commands:

duckdb -c "COPY (SELECT * FROM 'compounds.parquet') TO 'compounds.csv' (HEADER, DELIMITER ',');"

duckdb -c "COPY (SELECT * FROM 'patents.parquet') TO 'patents.csv' (HEADER, DELIMITER ',');"

duckdb -c "COPY (SELECT * FROM 'patent_compound_map.parquet') TO 'patent_compound_map.csv' (HEADER, DELIMITER ',');"

After that, take a look at the file sizes—you’ll see just how much more efficient Parquet really is.

Deprecation of old downloads

The legacy downloads are now officially deprecated:

• MAP files and compound data dumps: latest release from 2023
• Data client: last update on 20 June 2024

For more information check out the ChEMBL blog website. Here you will find a blog series detailing improvements to patent annotation, from full-text processing to image interpretation, and how the team plans to make this new content accessible to users.

Edit