From owner-chemistry@ccl.net Tue Mar 1 11:46:00 2022 From: "Rassolov, Vitaly RASSOLOV~!~mailbox.sc.edu" To: CCL Subject: CCL: Cloud backup in computational chemistry Message-Id: <-54611-220301093526-9376-6q54XbAhjGgBeRMKYwqxMA]*[server.ccl.net> X-Original-From: "Rassolov, Vitaly" Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_PH0PR19MB5147FFFF1880F2CED66E1C94EB029PH0PR19MB5147namp_" Date: Tue, 1 Mar 2022 14:35:15 +0000 MIME-Version: 1.0 Sent to CCL by: "Rassolov, Vitaly" [RASSOLOV+/-mailbox.sc.edu] --_000_PH0PR19MB5147FFFF1880F2CED66E1C94EB029PH0PR19MB5147namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Open Science Framework https://www.cos.io stores data at $4000/TB, guarante= ed for 50 years. I don't have any personal experience with it, and it seem= s pricey for an individual. If your institution is affiliated with the OSF= , perhaps the price is different. Center for Open Science At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress www.cos.io ________________________________ Vitaly Rassolov Department of Chemistry and Biochemistry University of South Carolina 631 Sumter St, Columbia SC 29208 https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_people= /directory/rassolov_vitaly.php ________________________________ > From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu[*]ccl.net on behalf of Andrew DeYoung andr= ewdaviddeyoung#%#gmail.com Sent: Monday, February 28, 2022 4:48 PM To: Rassolov, Vitaly Subject: CCL: Cloud backup in computational chemistry Hi, I have ~1.5 TB of analysis data on my home computer related to my PhD work,= which I completed last year. I would like to put this data in long-term s= torage in the cloud. I will retain a backup on an external hard drive, whi= ch I will keep at home. But I'm looking for an additional backup in anothe= r location (the other location being, ideally, the cloud). These files are= not themselves MD trajectories from my PhD, but rather are output and anal= yses related to the trajectories. (The trajectories themselves are tens of= TB and are being retained, for now, by my university.) Does anyone have experience with archiving scientific data on cloud storage= services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDrive? I realize this is the Computational Chemistry List, not the Information Tec= hnology List, but I'd be grateful for any insight you may have from the per= spective of a computational chemist. I will be paying for this myself, so the service needs to be reasonably pri= ced for a single user for ~5-10 TB or less. It's not immediately clear to = me whether all of these services are really available to individuals. I don't need fancy features like automatic backup, delta copying, and so fo= rth; I just need a place to safely store some files. I use Linux for runni= ng computation, but I'm not an IT expert by any means; I am willing to lear= n, though, especially when good documentation is available! And, clearly, = transferring ~1.5 TB to the cloud is itself a nontrivial task, but I'm will= ing to see what the options are. Here are some links to some seemingly popular services in my country (US), = but perhaps there are others more particularly suited for scientific data: AWS Amazon S3: https://aws.amazon.com/s3/ AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/ Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage iDrive: https://www.idrive.com/ Thank you for your time! Best, Andrew Andrew DeYoung, PhD Department of Chemistry Carnegie Mellon University --_000_PH0PR19MB5147FFFF1880F2CED66E1C94EB029PH0PR19MB5147namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed for 50 yea= rs.  I don't have any personal experience with it, and it seems pricey= for an individual.  If your institution is affiliated with the OSF, perhaps the price is different.
At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress
www.cos.io


Vitaly Rassolov
Department of Chemistry and Biochemistry
University of South Carolina
631 Sumter St, Columbia SC 29208
https://sc.edu/study/colleges_sc= hools/chemistry_and_biochemistry/our_people/directory/rassolov_vitaly.php
From: owner-chemistry+rasso= lov=3D=3Dmail.chem.sc.edu[*]ccl.net <owner-chemistry+rassolov=3D=3Dmail.ch= em.sc.edu[*]ccl.net> on behalf of Andrew DeYoung andrewdaviddeyoung#%#gmai= l.com <owner-chemistry[*]ccl.net>
Sent: Monday, February 28, 2022 4:48 PM
To: Rassolov, Vitaly <RASSOLOV[*]mailbox.sc.edu>
Subject: CCL: Cloud backup in computational chemistry
 
Hi, 

I have ~1.5 TB of analysis data on my home computer related to my= PhD work, which I completed last year.  I would like to put this data= in long-term storage in the cloud.  I will retain a backup on an exte= rnal hard drive, which I will keep at home.  But I'm looking for an additional backup in another location (the other locati= on being, ideally, the cloud).  These files are not themselves MD traj= ectories from my PhD, but rather are output and analyses related to the tra= jectories.  (The trajectories themselves are tens of TB and are being retained, for now, by my university.)

Does anyone have experience with archiving scientific data on clo= ud storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDr= ive?  

I realize this is the Computational Chemistry List, not the Informatio= n Technology List, but I'd be grateful for any insight you may have fr= om the perspective of a computational chemist.

I will be paying for this myself, so the service needs to be reasonabl= y priced for a single user for ~5-10 TB or less.  It's not immedi= ately clear to me whether all of these services are really a= vailable to individuals.

I don't need fancy features like automatic backup, delta copying, and = so forth; I just need a place to safely store some files.  I use Linux= for running computation, but I'm not an IT expert by any means; I am = willing to learn, though, especially when good documentation is available!  And, clearly, transferring ~1.5 TB = to the cloud is itself a nontrivial task, but I'm willing to see what = the options are.

Here are some links to some seemingly popular services in my country (= US), but perhaps there are others more particularly suited for scientific d= ata:

Thank you for your time!

Best,
Andrew

Andrew DeYoung, PhD
Department of Chemistry
Carnegie Mellon University
--_000_PH0PR19MB5147FFFF1880F2CED66E1C94EB029PH0PR19MB5147namp_-- From owner-chemistry@ccl.net Tue Mar 1 12:21:00 2022 From: "Anson Thomas thomasanson53##gmail.com" To: CCL Subject: CCL: Best way to visualize charge densities Message-Id: <-54612-220301110353-2007-SoPr1Opz9ie536QCa4oIkw|*|server.ccl.net> X-Original-From: Anson Thomas Content-Type: multipart/alternative; boundary="00000000000017962105d92a4fd9" Date: Tue, 1 Mar 2022 21:33:04 +0530 MIME-Version: 1.0 Sent to CCL by: Anson Thomas [thomasanson53%%gmail.com] --00000000000017962105d92a4fd9 Content-Type: text/plain; charset="UTF-8" Dear CCL members, I have obtained cube files of charge density difference using Quantum Espresso. I am trying to plot these as a contour plot in an aesthetically pleasing manner with good quality. I'm aware that XCrsyDen could do this, but the resolution of the exported file isn't good. Also tried Vesta, but was not satisfied with the results. I desire to plot something like figure 5 in this paper ( https://doi.org/10.1186/s11671-018-2605-3). Is there some other software/piece of code that can help accomplish similar results? Any help would be appreciated. Sincerely, Anson -- *Anson Thomas* M.Sc. Chemistry, IIT Roorkee https://www.linkedin.com/in/ansonthms/ --00000000000017962105d92a4fd9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Dear CCL members,
I have obtained cube files of charge density difference using = Quantum Espresso. I am trying to plot these as a contour plot in an aesthet= ically pleasing=C2=A0manner with=C2=A0good quality. I'm aware that XCrs= yDen could do this, but the=C2=A0resolution of the exported file isn't = good. Also tried Vesta,=C2=A0but was not satisfied with the results.
<= div>I desire to plot something like figure 5 in this paper (https://doi.org/10= .1186/s11671-018-2605-3). Is there some other software/piece of code th= at can help accomplish similar results?

Any help w= ould be appreciated.=C2=A0

Sincerely,
An= son



--
Anson Thomas
M.Sc. Chemistry, IIT Ro= orkee
--00000000000017962105d92a4fd9-- From owner-chemistry@ccl.net Tue Mar 1 13:33:00 2022 From: "Strahs, Dr. Daniel Bernard dstrahs]![pace.edu" To: CCL Subject: CCL: Cloud backup in computational chemistry Message-Id: <-54613-220301131613-23895-9OAcaAT89cTg1XSbnfP49A:server.ccl.net> X-Original-From: "Strahs, Dr. Daniel Bernard" Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_164615856515648810paceedu_" Date: Tue, 1 Mar 2022 18:16:05 +0000 MIME-Version: 1.0 Sent to CCL by: "Strahs, Dr. Daniel Bernard" [dstrahs+/-pace.edu] --_000_164615856515648810paceedu_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Rather than cloud, may I suggest a 2 TB Raid system? They can be had for le= ss than $200, and RAID 1 would provide adequate long-term stability. Dan Strahs ________________________________ > From: owner-chemistry+dstrahs=3D=3Dpace.edu---ccl.net on behalf of Rassolov, Vitaly RASSOLOV~!~mailbox.s= c.edu Sent: Tuesday, March 1, 2022 9:35 AM To: Strahs, Dr. Daniel Bernard Subject: CCL: Cloud backup in computational chemistry Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed fo= r 50 years. I don't have any personal experience with it, and it seems pri= cey for an individual. If your institution is affiliated with the OSF, per= haps the price is different. Center for Open Science At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress www.cos.io ________________________________ Vitaly Rassolov Department of Chemistry and Biochemistry University of South Carolina 631 Sumter St, Columbia SC 29208 https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_people= /directory/rassolov_vitaly.php ________________________________ > From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net on behalf of Andrew DeYoung = andrewdaviddeyoung#%#gmail.com Sent: Monday, February 28, 2022 4:48 PM To: Rassolov, Vitaly Subject: CCL: Cloud backup in computational chemistry Hi, I have ~1.5 TB of analysis data on my home computer related to my PhD work,= which I completed last year. I would like to put this data in long-term s= torage in the cloud. I will retain a backup on an external hard drive, whi= ch I will keep at home. But I'm looking for an additional backup in anothe= r location (the other location being, ideally, the cloud). These files are= not themselves MD trajectories from my PhD, but rather are output and anal= yses related to the trajectories. (The trajectories themselves are tens of= TB and are being retained, for now, by my university.) Does anyone have experience with archiving scientific data on cloud storage= services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDrive? I realize this is the Computational Chemistry List, not the Information Tec= hnology List, but I'd be grateful for any insight you may have from the per= spective of a computational chemist. I will be paying for this myself, so the service needs to be reasonably pri= ced for a single user for ~5-10 TB or less. It's not immediately clear to = me whether all of these services are really available to individuals. I don't need fancy features like automatic backup, delta copying, and so fo= rth; I just need a place to safely store some files. I use Linux for runni= ng computation, but I'm not an IT expert by any means; I am willing to lear= n, though, especially when good documentation is available! And, clearly, = transferring ~1.5 TB to the cloud is itself a nontrivial task, but I'm will= ing to see what the options are. Here are some links to some seemingly popular services in my country (US), = but perhaps there are others more particularly suited for scientific data: AWS Amazon S3: https://aws.amazon.com/s3/ AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/ Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage iDrive: https://www.idrive.com/ Thank you for your time! Best, Andrew Andrew DeYoung, PhD Department of Chemistry Carnegie Mellon University --_000_164615856515648810paceedu_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Rather than cloud, may I suggest a 2 TB Raid system? They can be had for= less than $200, and RAID 1 would provide adequate long-term stability.


Dan Strahs


From: owner-chemistry+= ;dstrahs=3D=3Dpace.edu---ccl.net <owner-chemistry+dstrahs=3D=3Dpace.ed= u---ccl.net> on behalf of Rassolov, Vitaly RASSOLOV~!~mailbox.sc.edu <owner-chemistry---ccl.net>
Sent: Tuesday, March 1, 2022 9:35 AM
To: Strahs, Dr. Daniel Bernard
Subject: CCL: Cloud backup in computational chemistry
 
Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed for 50 years.&nb= sp; I don't have any personal experience with it, and it seems pricey for a= n individual.  If your institution is affiliated with the OSF, perhaps= the price is different.
At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress
www.cos.io


Vitaly Rassolov
Department of Chemistry and Biochemistry
University of South Carolina
631 Sumter St, Columbia SC 29208
https://sc.edu/study/college= s_schools/chemistry_and_biochemistry/our_people/directory/rassolov_vitaly.p= hp
From: owner-chemistry+= ;rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net <owner-chemistry+rassolov= =3D=3Dmail.chem.sc.edu=3Dccl.net> on behalf of Andrew DeYoung andrewdavi= ddeyoung#%#gmail.com <owner-chemistry=3Dccl.net>
Sent: Monday, February 28, 2022 4:48 PM
To: Rassolov, Vitaly <RASSOLOV=3Dmailbox.sc.edu>
Subject: CCL: Cloud backup in computational chemistry
 
Hi, 

I have ~1.5 TB of analysis data on my home computer related to my= PhD work, which I completed last year.  I would like to put this data= in long-term storage in the cloud.  I will retain a backup on an exte= rnal hard drive, which I will keep at home.  But I'm looking for an additional backup in another location (the other locati= on being, ideally, the cloud).  These files are not themselves MD traj= ectories from my PhD, but rather are output and analyses related to the tra= jectories.  (The trajectories themselves are tens of TB and are being retained, for now, by my university.)

Does anyone have experience with archiving scientific data on clo= ud storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDr= ive?  

I realize this is the Computational Chemistry List, not the Informatio= n Technology List, but I'd be grateful for any insight you may have fr= om the perspective of a computational chemist.

I will be paying for this myself, so the service needs to be reasonabl= y priced for a single user for ~5-10 TB or less.  It's not immedi= ately clear to me whether all of these services are really a= vailable to individuals.

I don't need fancy features like automatic backup, delta copying, and = so forth; I just need a place to safely store some files.  I use Linux= for running computation, but I'm not an IT expert by any means; I am = willing to learn, though, especially when good documentation is available!  And, clearly, transferring ~1.5 TB = to the cloud is itself a nontrivial task, but I'm willing to see what = the options are.

Here are some links to some seemingly popular services in my country (= US), but perhaps there are others more particularly suited for scientific d= ata:


Thank you for your time!

Best,
Andrew

Andrew DeYoung, PhD
Department of Chemistry
Carnegie Mellon University
--_000_164615856515648810paceedu_-- From owner-chemistry@ccl.net Tue Mar 1 14:28:01 2022 From: "Lalith Perera lalithmperera**gmail.com" To: CCL Subject: CCL: Best way to visualize charge densities Message-Id: <-54614-220301141630-23430-af2QHl5My/Co8vNXzGQpcA(a)server.ccl.net> X-Original-From: Lalith Perera Content-Type: multipart/alternative; boundary="0000000000003fb3c705d92d0067" Date: Tue, 1 Mar 2022 14:16:13 -0500 MIME-Version: 1.0 Sent to CCL by: Lalith Perera [lalithmperera**gmail.com] --0000000000003fb3c705d92d0067 Content-Type: text/plain; charset="UTF-8" Anson, You can try Multiwfn to create such figures. This is an open-source package. http://sobereva.com/multiwfn/ Good luck. -Lalith On Tue, Mar 1, 2022 at 2:02 PM Anson Thomas thomasanson53##gmail.com < owner-chemistry#ccl.net> wrote: > Dear CCL members, > > I have obtained cube files of charge density difference using Quantum > Espresso. I am trying to plot these as a contour plot in an aesthetically > pleasing manner with good quality. I'm aware that XCrsyDen could do this, > but the resolution of the exported file isn't good. Also tried Vesta, but > was not satisfied with the results. > I desire to plot something like figure 5 in this paper ( > https://doi.org/10.1186/s11671-018-2605-3). Is there some other > software/piece of code that can help accomplish similar results? > > Any help would be appreciated. > > Sincerely, > Anson > > > > -- > *Anson Thomas* > M.Sc. Chemistry, IIT Roorkee > https://www.linkedin.com/in/ansonthms/ > -- Work: Home: MD F0-08 NIEHS/NIH 502 Rock Garden Road 111 T.W. Alexander Dr. P.O. Box 12233 Chapel Hill, NC 27516 RTP NC 27709-2233 Phone: (919) 967 8387 Phone: (984)287 3375 e-mail: Lalith.Perera#nih.gov lalithMperera#gmail.com --0000000000003fb3c705d92d0067 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Anson,
You can try Multiwfn to create such figur= es. This is an open-source package.=C2=A0
Good luck.
-Lalith

On Tue, Mar 1, 2022 at 2:02 PM Anson Thomas thomasanson53##= gmail.com <owner-chemistry#ccl.net> wrote:
Dear CCL members,

I have obtained cube fil= es of charge density difference using Quantum Espresso. I am trying to plot= these as a contour plot in an aesthetically pleasing=C2=A0manner with=C2= =A0good quality. I'm aware that XCrsyDen could do this, but the=C2=A0re= solution of the exported file isn't good. Also tried Vesta,=C2=A0but wa= s not satisfied with the results.
I desire to plot something like= figure 5 in this paper (https://doi.org/10.1186/s11671-018-2605-3). Is th= ere some other software/piece of code that can help accomplish similar resu= lts?

Any help would be appreciated.=C2=A0

Sincerely,
Anson



--
Anso= n Thomas
M.Sc. Chemistry, IIT Roorkee


--
Work: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Home:
= =C2=A0 =C2=A0MD F0-08=C2=A0=C2=A0 NIEHS/NIH=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0502 Rock Garden Road =C2=A0 =C2=A0 =C2=A0
=C2=A0 =C2=A0111 T.W. Alex= ander Dr.=C2=A0=C2=A0 P.O. Box 12233 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Chapel Hill, =C2=A0NC 27516
=C2= =A0 =C2=A0RTP =C2=A0NC 27709-2233 =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Phone: (919) 967 8387
=C2=A0=C2=A0 Phone: (984)287 3375
=
=C2=A0 =C2=A0e-mail: Lalith.Perera#nih.gov =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0lalithMperera#gmail.com
--0000000000003fb3c705d92d0067-- From owner-chemistry@ccl.net Tue Mar 1 15:57:01 2022 From: "Detlev Conrad Mielczarek detlevcm#,#googlemail.com" To: CCL Subject: CCL: Cloud backup in computational chemistry Message-Id: <-54615-220301153525-10600-f0z6AeGqObzIh223P2Vseg() server.ccl.net> X-Original-From: Detlev Conrad Mielczarek Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=utf-8 Date: Tue, 01 Mar 2022 21:35:15 +0100 MIME-Version: 1.0 Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com] Please note, RAID is NOT a backup. The topic is fairly complex if done rigorously. Look up "bit rot" - data life is finite too. Look up the risk of data corruption on a RAID system - or loss with say malware or error. The topic is not simple - unfortunately. And then there is data lifetime on disks which is not necessarily a resolved question. Tape is considered the best for long term storage at current - maybe archive quality DVDs. Detlev Sent from my BlackBerry   Original Message   > From: owner-chemistry()ccl.net Sent: 1 March 2022 21:19 To: detlevcm()googlemail.com Reply to: chemistry()ccl.net Subject: CCL: Cloud backup in computational chemistry Rather than cloud, may I suggest a 2 TB Raid system? They can be had for less than $200, and RAID 1 would provide adequate long-term stability. Dan Strahs > From: owner-chemistry+dstrahs==pace.edu##ccl.net on behalf of Rassolov, Vitaly RASSOLOV~!~mailbox.sc.edu Sent: Tuesday, March 1, 2022 9:35 AM To: Strahs, Dr. Daniel Bernard Subject: CCL: Cloud backup in computational chemistry   Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed for 50 years.  I don't have any personal experience with it, and it seems pricey for an individual.  If your institution is affiliated with the OSF, perhaps the price is different. Center for Open Science At the Center for Open Science, our mission is to increase openness, integrity, and reproducibility of scholarly research. Promoting these practices within the research funding and publishing communities accelerates scientific progress www.cos.io Vitaly Rassolov Department of Chemistry and Biochemistry University of South Carolina 631 Sumter St, Columbia SC 29208 https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_people/directory/rassolov_vitaly.php > From: owner-chemistry+rassolov==mail.chem.sc.edu=ccl.net on behalf of Andrew DeYoung andrewdaviddeyoung#%#gmail.com Sent: Monday, February 28, 2022 4:48 PM To: Rassolov, Vitaly Subject: CCL: Cloud backup in computational chemistry   Hi,  I have ~1.5 TB of analysis data on my home computer related to my PhD work, which I completed last year.  I would like to put this data in long-term storage in the cloud.  I will retain a backup on an external hard drive, which I will keep at home.  But I'm looking for an additional backup in another location (the other location being, ideally, the cloud).  These files are not themselves MD trajectories from my PhD, but rather are output and analyses related to the trajectories.  (The trajectories themselves are tens of TB and are being retained, for now, by my university.) Does anyone have experience with archiving scientific data on cloud storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDrive?   I realize this is the Computational Chemistry List, not the Information Technology List, but I'd be grateful for any insight you may have from the perspective of a computational chemist. I will be paying for this myself, so the service needs to be reasonably priced for a single user for ~5-10 TB or less.  It's not immediately clear to me whether all of these services are really available to individuals. I don't need fancy features like automatic backup, delta copying, and so forth; I just need a place to safely store some files.  I use Linux for running computation, but I'm not an IT expert by any means; I am willing to learn, though, especially when good documentation is available!  And, clearly, transferring ~1.5 TB to the cloud is itself a nontrivial task, but I'm willing to see what the options are. Here are some links to some seemingly popular services in my country (US), but perhaps there are others more particularly suited for scientific data: AWS Amazon S3: https://aws.amazon.com/s3/ AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/ Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage iDrive: https://www.idrive.com/ Thank you for your time! Best, Andrew Andrew DeYoung, PhD Department of Chemistry Carnegie Mellon University From owner-chemistry@ccl.net Tue Mar 1 23:33:01 2022 From: "Abhik Seal abhik1368*o*gmail.com" To: CCL Subject: CCL: Cloud backup in computational chemistry Message-Id: <-54616-220301180642-18620-wZCSe3q/bcSLCb0MMsUDxQ(~)server.ccl.net> X-Original-From: Abhik Seal Content-Type: multipart/alternative; boundary="0000000000007b8c0e05d9303767" Date: Tue, 1 Mar 2022 17:06:24 -0600 MIME-Version: 1.0 Sent to CCL by: Abhik Seal [abhik1368*gmail.com] --0000000000007b8c0e05d9303767 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable One way Would be to reduce the file size such that you don=E2=80=99t need h= igh amount of space and also reduces cost . Python blosc is a high performance compressor . Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. Another one is feather format in Apache arrow . Hope you get some ideas . Abhij On Tue, Mar 1, 2022 at 3:47 PM Detlev Conrad Mielczarek detlevcm#,# googlemail.com wrote: > > Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com] > Please note, RAID is NOT a backup. > > The topic is fairly complex if done rigorously. Look up "bit rot" - data > life is finite too. > Look up the risk of data corruption on a RAID system - or loss with say > malware or error. > > The topic is not simple - unfortunately. > And then there is data lifetime on disks which is not necessarily a > resolved question. > Tape is considered the best for long term storage at current - maybe > archive quality DVDs. > > Detlev > > > > Sent from my BlackBerry > > > > Original Message > > > > From: owner-chemistry[*]ccl.net > Sent: 1 March 2022 21:19 > To: detlevcm[*]googlemail.com > Reply to: chemistry[*]ccl.net > Subject: CCL: Cloud backup in computational chemistry > > > Rather than cloud, may I suggest a 2 TB Raid system? They can be had for > less than $200, and RAID 1 would provide adequate long-term stability. > > > Dan Strahs > > > > > > From: owner-chemistry+dstrahs=3D=3Dpace.edu##ccl.net > > on behalf of Rassolov, Vitaly RASSOLOV~!~ > mailbox.sc.edu > Sent: Tuesday, March 1, 2022 9:35 AM > To: Strahs, Dr. Daniel Bernard > Subject: CCL: Cloud backup in computational chemistry > > Open Science Framework https://www.cos.io stores data at $4000/TB, > guaranteed for 50 years. I don't have any personal experience with it, a= nd > it seems pricey for an individual. If your institution is affiliated wit= h > the OSF, perhaps the price is different. > > Center for Open Science > At the Center for Open Science, our mission is to increase openness, > integrity, and reproducibility of scholarly research. Promoting these > practices within the research funding and publishing communities > accelerates scientific progress > www.cos.io > > > > > Vitaly Rassolov > Department of Chemistry and Biochemistry > University of South Carolina > 631 Sumter St, Columbia SC 29208 > > https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_peop= le/directory/rassolov_vitaly.php > > > From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net > on behalf of A= ndrew > DeYoung andrewdaviddeyoung#%#gmail.com > Sent: Monday, February 28, 2022 4:48 PM > To: Rassolov, Vitaly > Subject: CCL: Cloud backup in computational chemistry > > Hi, > > > I have ~1.5 TB of analysis data on my home computer related to my PhD > work, which I completed last year. I would like to put this data in > long-term storage in the cloud. I will retain a backup on an external ha= rd > drive, which I will keep at home. But I'm looking for an additional back= up > in another location (the other location being, ideally, the cloud). Thes= e > files are not themselves MD trajectories from my PhD, but rather are outp= ut > and analyses related to the trajectories. (The trajectories themselves a= re > tens of TB and are being retained, for now, by my university.) > > > Does anyone have experience with archiving scientific data on cloud > storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or > iDrive? > > > I realize this is the Computational Chemistry List, not the Information > Technology List, but I'd be grateful for any insight you may have from th= e > perspective of a computational chemist. > > > I will be paying for this myself, so the service needs to be reasonably > priced for a single user for ~5-10 TB or less. It's not immediately clea= r > to me whether all of these services are really available to individuals. > > > I don't need fancy features like automatic backup, delta copying, and so > forth; I just need a place to safely store some files. I use Linux for > running computation, but I'm not an IT expert by any means; I am willing = to > learn, though, especially when good documentation is available! And, > clearly, transferring ~1.5 TB to the cloud is itself a nontrivial task, b= ut > I'm willing to see what the options are. > > > Here are some links to some seemingly popular services in my country (US)= , > but perhaps there are others more particularly suited for scientific data= : > > > AWS Amazon S3: https://aws.amazon.com/s3/ > AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/ > Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage > > iDrive: https://www.idrive.com/ > > > Thank you for your time! > > > Best, > Andrew > > Andrew DeYoung, PhD > Department of Chemistry > Carnegie Mellon University > > > > -=3D This is automatically added to each message by the mailing script = =3D-> > > -- Cheers, Abhik Seal Ph.D. (Cheminformatics) --0000000000007b8c0e05d9303767 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
One way Would be to reduce the file size such that you do= n=E2=80=99t need high amount of space and also reduces cost . Python blosc = is a high performance compressor .=C2=A0Blosc works well for compressing numeric= al arrays that contains data with relatively low entropy, like sparse data,= time series, grids with regular-spaced values, etc.

Another one is feather format = in Apache arrow . Hope you get some ideas .

Abhij



On Tue, Mar 1, 2022 at 3:47 PM Detlev Conrad Mielczarek detlevcm#,#googlemail.com <owner-chemistry^ccl.net> wrote:

Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com]
Please note, RAID is NOT a backup.

The topic is fairly complex if done rigorously. Look up "bit rot"= - data life is finite too.
Look up the risk of data corruption on a RAID system - or loss with say mal= ware or error.

The topic is not simple - unfortunately.
And then there is data lifetime on disks which is not necessarily a resolve= d question.
Tape is considered the best for long term storage at current - maybe archiv= e quality DVDs.

Detlev



Sent from my BlackBerry



=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Original Message =C2=A0=C2=A0 =C2=A0

> From: owner-chemistry[*]ccl.net
Sent: 1 March 2022 21:19
To: detlevcm[*]googlemail.com
Reply to: chemistry[*]ccl.net
Subject: CCL: Cloud backup in computational chemistry


Rather than cloud, may I suggest a 2 TB Raid system? They can be had for le= ss than $200, and RAID 1 would provide adequate long-term stability.


Dan Strahs




> From: owner-chemistry+dstrahs=3D=3Dpace.edu##ccl.net <owner-ch= emistry+dstrahs=3D=3Dpace.edu##ccl.net> on behalf of Rassolov, Vita= ly RASSOLOV~!~mailbox.sc.edu<owner-chemistry##ccl.net>
Sent: Tuesday, March 1, 2022 9:35 AM
To: Strahs, Dr. Daniel Bernard
Subject: CCL: Cloud backup in computational chemistry
=C2=A0
Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed = for 50 years.=C2=A0 I don't have any personal experience with it, and i= t seems pricey for an individual.=C2=A0 If your institution is affiliated w= ith the OSF, perhaps the price is different.

Center for Open Science
At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress
www.cos.= io




Vitaly Rassolov
Department of Chemistry and Biochemistry
University of South Carolina
631 Sumter St, Columbia SC 29208
https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_p= eople/directory/rassolov_vitaly.php

> From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net <owner-ch= emistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net> on behalf of Andrew DeYou= ng andrewdaviddeyoung#%#gmail.com<owner-chemistry=3Dccl.net>
Sent: Monday, February 28, 2022 4:48 PM
To: Rassolov, Vitaly <RASSOLOV=3Dmailbox.sc.edu>
Subject: CCL: Cloud backup in computational chemistry
=C2=A0
Hi,=C2=A0


I have ~1.5 TB of analysis data on my home=C2=A0computer related to my PhD = work, which I completed last year.=C2=A0 I would like to put this data in l= ong-term storage in the cloud.=C2=A0 I will retain a backup on an external = hard drive, which I will keep at home.=C2=A0 But I'm looking for an add= itional backup in another location (the other location being, ideally, the = cloud).=C2=A0 These files are not themselves MD trajectories from my PhD, b= ut rather are output and analyses related to the trajectories.=C2=A0 (The t= rajectories themselves are tens of TB and are being retained, for now, by m= y university.)


Does anyone have experience with archiving scientific=C2=A0data on cloud st= orage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDrive?= =C2=A0=C2=A0


I realize this is the Computational Chemistry List, not the Information Tec= hnology List, but I'd be grateful for any insight you=C2=A0may have fro= m the perspective of a computational chemist.


I will be paying for this myself, so the service needs to be reasonably pri= ced for a single user for=C2=A0~5-10 TB or less.=C2=A0 It's not immedia= tely clear to me=C2=A0whether=C2=A0all of=C2=A0these services are really av= ailable to individuals.


I don't need fancy features like automatic backup, delta copying, and s= o forth; I just need a place to safely store some files.=C2=A0 I use Linux = for running computation, but I'm not an IT=C2=A0expert by any means; I = am willing to learn, though, especially when good documentation is availabl= e!=C2=A0 And, clearly, transferring ~1.5 TB to the cloud is itself a nontri= vial=C2=A0task, but I'm willing to see what the options are.


Here are some links to some seemingly popular services in my country (US), = but perhaps there are others more particularly suited for scientific data:<= br>

AWS Amazon S3:=C2=A0https://aws.amazon.com/s3/
AWS Amazon S3 Glacier:=C2=A0https://aws.amazon.com= /s3/storage-classes/glacier/
Microsoft Azure:=C2=A0https://azure.microsoft.com= /en-us/services/#storage

iDrive:=C2=A0https://www.idrive.com/


Thank you for your time!


Best,
Andrew

Andrew DeYoung, PhD
Department of Chemistry
Carnegie Mellon University



-=3D This is automatically added to each message by the mailing script =3D-=
E-mail to subscribers: CHEMISTRY^ccl.net or use:
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/cgi-bin/ccl/s= end_ccl_message

E-mail to administrators: CHEMISTRY-REQUEST^ccl.net or use
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/cgi-bin/ccl/s= end_ccl_message
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/chemistry/sub_un= sub.shtml

Before posting, check wait time at: http://www.ccl.net

Job: http://www.ccl.net/jobs
Conferences: http://server.ccl.net/chemist= ry/announcements/conferences/

Search Messages: http://www.ccl.net/chemistry/sear= chccl/index.shtml
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/spammers.txt

RTFI: http://www.ccl.net/chemistry/aboutccl/ins= tructions/


--
<= div dir=3D"ltr">
=C2=A0
Cheers,
Abhik Seal =C2=A0Ph.D. (Cheminformatics)

--0000000000007b8c0e05d9303767--