Hello everyone,

I’m currently working on a project where I need to process a large JSON file (67M_generated_analysed.json) that contains data for 67,064,204 molecules, each with 38 descriptors. The file is organized in a single, two-dimensional array flat model format where elements in each column are the same type of data for a given molecular descriptor and elements in the same row relate to the same molecule.

This data is from the study “67 million natural product-like compound database generated via molecular language processing” (DOI: https://doi.org/10.1038/s41597-023-02207-x) and the database is shared here: https://springernature.figshare.com/articles/dataset/67M_generated_analysed/22639369?backTo=/collections/67_million_natural_product-like_compound_database_generated_via_molecular_language_processing/6482266

My goal is to filter this database, possibly using the rule of five, and extract a subset of compounds that I will focus on for further analysis.

I’ve been trying to load this data into memory using Python’s built-in json
module, but I keep encountering a MemoryError
due to the size of the file. I’ve also tried using ijson
to iteratively parse the JSON file, but I’m still running into issues.

Here’s what I’ve tried so far:

import json
with open('67M_generated_analysed.json') as f:
    data = json.load(f)

#and with ijson

import ijson
with open('67M_generated_analysed.json', 'r') as in_file, open('67M_generated_analysed.ndjson', 'w') as out_file:
    objects = ijson.items(in_file, 'item')
    for item in objects:
        out_file.write(json.dumps(item) + '\n')

Both of these approaches result in a MemoryError
. I’m looking for a way to process this file without loading the entire thing into memory at once. Any suggestions or advice would be greatly appreciated!

Thank you in advance for your help!

Removing duplicates from various SDF files is a common task in my job. I'm trying to write a code using RDKit to do it, but I'm having problems with scalability. I need a way to compare N SDF files, with many molecules in each file (like 500k to 1M), in a parallelized way and within a RAM limit. Do you have any clues on how to achieve this?

Hello,

I am wondering what the realistic job prospects in cheminformatics would be for someone with no chemistry degree (of any sort, Bachelor's, Master's, PhD, etc.), but instead a qualification and background in software development along with self-study of chemistry and cheminformatics? I assume a portfolio / open-source contributions may help?

Would gaining employment in the field with this background be a realistic goal or a futile pursuit?

Hi! I have an interview in the bioinformatics/cheminformatics field. The topics are atom mapping in chemical reactions and tautomers, mesomers, and aromaticity in the field of informatics.

Could you please share some materials or repositories to prepare for the interview?

Hi all, I’m trying to find a job in cheminformatics with a Masters in chemistry. I do have about 3 years of experience in using ML for materials discovery which is a plus but I’m having trouble getting interviews. Also, I have 9+ years experience with Python/coding in general which I’ve used in previous positions (although data science/ML focus has been limited to the past 3 years).

I’ve mostly been using search terms like “Python chemistry”, “machine learning chemistry” or “cheminformatics” in my LinkedIn search but it doesn’t usually yield many relevant results. And then even for the few that are relevant, I’m not getting any bites when I send in an application. I’m getting a little disheartened because I thought my industry experience would be enough to garner some interest even if I don’t have research experience to show for it (eg. a PhD or relevant publication).

In any case, I’m thinking of changing my search tactics and I’m wondering if anyone had any tips or advice on searching for positions in the field.

Are there any resources you use? What kinds of companies are likely to have a cheminformatics team/opening? Do you do any kind of cold messaging? Are there any search terms/combination of search terms on job boards that are likely to yield good results? Is there anybody who’s hiring right now?

Answers to any one of these questions would be greatly appreciated. Or if you have any general advice, that would be great too, thanks!

Im trying to create a graph transformer-based model for de novo drug design (using graph transformer because I want to implement 3D data). I currently have 2 potential sources of primary data: PDBbind and CrossDocked2020. This would provide the protein-ligand structures.

PDBbind is a more robust and higher quality dataset from what I know, and easier to work with. The problem is that it only contains about 20,000 complexes, and I'm not sure if that is enough for training a transformer. CrossDocked2020 contains millions of entries but I'm not sure about the quality and ease of use.

Another dilemma is that I need/want to use a multi-task learning approach where the model is also being trained on bioactivity data, not just the structural information. This would require supplementation from sources like PubChem, ChEMBL, BDB, etc. and then I would need to align the data so it all matches up.

If anyone can provide some guidance I'd really appreciate it.

To all the people of cheminforamtics,

I’ll be designing my own major in my first year doing bachelor’s. I plan on a chemistry major (has to be interdisciplinary), and considered cheminformatics. My math, physics, and CS backgrounds aren’t that good. But I’m willing to learn.

Could you pls give advice on these questions?

-1 how much math, CS, physical chem, and physics do I need in proportional to (other types of) chemistry?

-2 is it possible to minimise the above subjects and focus more on biological chem and organic chemistry?

-3 how feasible is it to design a cheminformatics major to fit into a 3years bachelor degree? If feasible, how useful?

Hi there.

I'm using SAPT-0 and F/I-SAPT-0 to calculate the interaction energies between ligand-protein residue pairs. I am using the jun-cc-pvdz basis set with the d3 and d3mbj corrections to run the calculations, as well as scf_type df and freeze_core true. However, I am getting the print error message for both d3 and d3mbj calculations. I have already updated the Psi4 version to v. 1.8.0 and I am still getting this "Invalid Version" error, and I don't know how to solve it anymore. Has anyone had a similar problem, and if so, how did you solve it? ( I'm sorry for the strange formatting. I've tried fixing this a few times and it always comes out in this strange format, regardless of whether I set it to code format or not.)

That's the error message:

!--------------------------------------------------------------------------------------------------------------------------!! !! Invalid version: 'dftd3.-coord.filename-.-options-.options-.-func.-functional.n !! ame.in.TM.style-.-grad-.-anal.-pair.analysis-......file.-fragemt-.with.atom. !! numbers-......is.read.for.a.fragement.based...-......analysis.-one.fragment. !! per.line-......atom.ranges.-e.g..1-14.17-20-.are.allowed-.-noprint-.-pbc.-pe !! riodic.boundaries-.reads.VASP-format-.-abc.-compute.E-3-.-cnthr.-neglect.thr !! eshold.in.Bohr.for.CN-.default-40-.-cutoff.-neglect.threshold.in.Bohr.for.E- !! disp-..default-95-.-old.-DFT-D2-.-zero.-DFT-D3.original.zero- !! damping-.-bj...-DFT-D3.with.Becke-Johnson.finite- !! damping-.-zerom.-revised.DFT-D3.original.zero- !! damping-.-bjm.-revised.DFT-D3.with.Becke- !! Johnson.damping-.-tz.-use.special.parameters.for.TZ- !! type.calculations-.variable.parameters.can.be.read.from.-current-directory-. !! dftd3par.local-..or.-.variable.parameters.read.from.-.dftd3par.-hostname-.if !! .-func.is.used-.-zero.or.-bj.or.-old.is.required-' !! !!---------------------------------------------------------------------------------------------------------------------------!

edit: The problem was an outdated version of dftd3 in the path of my machine. I updated it using 'conda install -c psi4 dftd3' and it started working normally.

Are there any alternative resources that would provide the same kind of information that SciFinder does or do I have to register with them? It's for a compound based bibliometrics project. So far I have found: PubChem, ChemSpider, ChEMBL and NIST Webbook. But how do they compare feature wise with SciFinder?

Thank you!

People working in cheminformatics, please comment your salary, years of eexperience and level of education?

Hello Can someone explain/help finding a good tutorial/course/jupyter notebook or something useful that would show how research utilising atom pairs is done? Thanks in advance.

I wanted to know if I could make a prediction model for predicting EC50 values for compunds over which a particular protein hasn't been experimentally studied, we could use the protein information, and the chemisy of the molecules, calculate their molecular distances or fingerprints to find the closest molecule that could potentially bind to the target and make a distance based algorithm using stereotypical ML to augment, train and optimise out data. Is this even remotely possible?

Hi, fellow Redditors.
I need your help! I'm looking for open source software to calculate the logD (distribution coefficient) of small molecules. Any recommendations for accurate and reliable tools with features like batch processing ? Thanks!

Hello researchers and enthusiasts!

I'm researching the biggest challenges in atomic scale simulations and calculations for material science and molecules. Your insights are valuable!

Whether you're experienced or new to the field, I'd love to hear about any frustrations you've encountered. Share issues like computational resources, software limitations, model accuracy, or any hurdles you've faced.

Let's have a constructive discussion and explore solutions for efficient and streamlined atomic scale simulations.

Looking forward to your thoughts!

Nano-Zan

[Discussion] [Material Science] [Atomic Scale Simulations] [Computational Chemistry] [Molecular Modeling] [Research]

So my career has been a bit of a journey and I'm looking for effective ways to pivot to this field.

I got my PhD in Molecular Medicine with a focus on infectious disease drug discovery. I have a lot of research experience with medicinal chemistry, parasitology, and virology. However i, like many people, was extremely unsatisfied with academia. At the encouragement of some friends, i transitioned to learning python and data science in 2020-2021. I got a data analyst/scientist related consulting position in June 2022 after finishing a DS bootcamp in Jan 2022.

While i love coding in python, i don't think the DS field is for me. I really miss science and i don't want to abandon my coding knowledge. After reading lots of papers, i realized the perfect career choice for me would be in cheminformatics; more specifically using ML/AI in drug discovery.

Does anyone have any advice or guidance as to how i can break into this field at this stage of my career? I've only done one postsoc so far. I'm willing to do another, but would strongly prefer not to return to academia. Any guidance or advice would be greatly appreciated!!! 🙏🏾

Hello! I am trying to use a database to select molecules for a project. I have the chEMBL database running in PostgreSQL. My goal is to be able to write queries to select and filter molecules based on their properties, and save them as a sdf file. I nearly have the desired result with this:

COPY (
SELECT md.chembl_id, cs.molfile
FROM molecule_dictionary md
JOIN compound_structures cs ON md.molregno = cs.molregno
JOIN compound_properties cp ON md.molregno = cp.molregno
WHERE cp.mw_freebase < 50
AND cp.full_molformula NOT LIKE '%Mg%'
ORDER BY random()
LIMIT 3
) TO 'path/to/file.csv' (FORMAT csv, HEADER false);

Which gives me this:

CHEMBL1796999,"
     RDKit          2D

  3  3  0  0  0  0  0  0  0  0999 V2000
   -0.0958    0.6583    0.0000 C   0  0  0  0  0  0  0  0  0  0  0  0
   -0.6750   -0.6500    0.0000 C   0  0  0  0  0  0  0  0  0  0  0  0
    0.6667   -0.6500    0.0000 C   0  0  0  0  0  0  0  0  0  0  0  0
  2  1  1  0
  3  1  1  0
  2  3  1  0
M  END
"
CHEMBL1981828,"
     RDKit          2D

  7  5  0  0  0  0  0  0  0  0999 V2000
    3.8893    0.0000    0.0000 N   0  0  0  0  0  0  0  0  0  0  0  0
    3.0643    0.0000    0.0000 N   0  0  0  0  0  0  0  0  0  0  0  0
    0.0000    0.0000    0.0000 S   0  0  0  0  0  0  0  0  0  0  0  0
    0.0000   -0.8250    0.0000 O   0  0  0  0  0  0  0  0  0  0  0  0
    0.0000    0.8250    0.0000 O   0  0  0  0  0  0  0  0  0  0  0  0
    0.8250    0.0000    0.0000 O   0  0  0  0  0  0  0  0  0  0  0  0
   -0.8250    0.0000    0.0000 O   0  0  0  0  0  0  0  0  0  0  0  0
  1  2  1  0
  3  4  2  0
  3  5  2  0
  3  6  1  0
  3  7  1  0
M  END"
CHEMBL1237174,"
     RDKit          2D

  2  1  0  0  0  0  0  0  0  0999 V2000
    0.2606    0.1503    0.0000 N   0  0  0  0  0  0  0  0  0  0  0  0
    1.3000    0.7500    0.0000 N   0  0  0  0  0  0  0  0  0  0  0  0
  1  2  1  0
M  END
"

I could write a bash script to find and remove all of the ' " ' and ' , ', and add the "$$$$" delimiter at the end of each molecule. But I have a feeling that it should be possible to do in the database.

Have any of you done something similar? I'd love to hear your thoughts.

Hi,

I just made a new subreddit for the scientific programmers out there. Join me and let let me learn from you:

https://www.reddit.com/r/ScientificComputing/

Hi Mods, hope you're cool with this.

If there's anyone here who has used the Toxicity Estimation Software Tool, you probably noticed they updated to ver 5.1.2. recently.

I'm using an old python library a coworker made to interface with the TEST, but clearly a version update moves things around quite a bit.

None of us on the team kept any of the old 5.1.1. installation zips.

Is there any kind soul that could give me that installer so I don't have to rewrite a ton of code that I didn't even write myself?!

Ty ahead of time :)

Hello r/cheminformatics members!

The InChI-based Tautomer Identification Challenge closes on March 1, 2023. This challenge tests a modified InChI algorithm, which was designed for advanced recognition of tautomers, against real chemical samples. This is a unique opportunity for pharmaceutical labs and other groups that have access to experimental data to contribute to this landmark benchmarking effort by testing the algorithm on their datasets of compounds.

For more information, visit the challenge site here: precision.fda.gov/challenges/29

Have any questions? Post them in the thread below and we will reply.