【免费】Cdo.rbpy.ProjectManagementService.pdf资源-CSDN文库

需积分: 0 7 浏览量 2019-11-07 15:44:28 上传评论收藏 751KB PDF 举报

资源推荐

资源详情

资源评论

2019/11/4 Cdo{rbpy}CDOProjectManagementService
https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D 1/7
目
录
Cdo{rb,py}
Whythe...
whatitis(not)...
Usage
online/offlinehelp
IO
Options
OperatorChains
SpecialFeatures
Tempfilehan dling
Con ditionalProcessing
MultithreaddingRuby
MultithreaddingPython
Dataaccessvianumpy/narray/xarray/cdf
Writeyourownoperators
Futureversions
Installation
Ruby
Python
ForMPIMETusers
Cdo{rb,py}
Why the ...
You'rehappywithusingCDOinashell?You'rehesitanttouseascriptinglanguageinstead,becausesh ellsaresoeasy
touse?Herearesomecoolfeature s,you'llh ar dlygetwith inashellcontext:
Directdataaccessvianum py/narray
Ifyouhavepythonnetcdf4orscipyinstalled,youwillhavedirectaccesstothevaluesofyourfields.Pythonand
Rubyofferarichsetofscientificlibrariestoworkwiththesedataarrays,e.g.plotting.
Automatictempfilehandling
Whendealingwithtemporarydata,youwillneverhavetodothingsmanually.Nocleanup,nomanualcreation
andrenamingoffiles.Forwhateveryouwanttosave,givearealoutputfilenametherestisdone
automatically.
Flexibleparallelization
Youhaveahugelongrunningjob,t hatcouldbedoneonparallel?Ingeneralshellsdonotofferenough
parallelismtohavefinegrainedcontrolthe&isnotenoughfor100to1000jobs,becauseyoursystemwill
slowdownverymuchandinaminute,you'llgetyourcallfromtheadministrators.
BothPythonandRubyofferwhatyoujustneed:Run1000routinecallswithonly12concu rrentprocessesor
threads.WithCdo{rb,py}younowcanm akeuseofit!
Conditionalprocessing
Avoidreprocessingiftheoutputfilesisalreadyondisk.Wiselyused,thiscanspeedupyourscript
tremendously.Thisbehaviourcanbeswitchedongloballyorjustforasinglecalls.E.g.whenanalysingoutput
datafromarunningexperimentsyoucanrunthesamescriptoverandoveragainandyou'llalwaysgetthe
latestresults.
writenewoperatorsoutofoldones
PythonandRubybindingsareopenlibraries,t hatcanbeextendedatruntime.Ifyouneedanewoperator,justwriteitinPythonorRuby!
Thereisa githubrepositoryforeasycodesharingandwherethe changelogistracked.
Ifyouhavequestions,pleaseusetheCDOforum.
what it is (not) ...
Thisscriptinglanguagepackageisessen tiallyawrapperaroundtheCDObinary.Itparsesmethodargumentsandoptions,buildsacommandlineandexecutesit.Thereis
nosharedlibrarybackendwhich callsCDOoperators.Thishassomeadvantages:
operatorchainingisfullysupported
multipleCDObinariescanbeusedatthesametimeusingsetCdo()( alt.link)
packagesarehighlyportable,becausetheyarepurepython/rubyimplementations
Usage
Almostallfeaturesarecoveredbyunitstests.Thesesshouldbeagoodstartingpointtogetanimpressioninhowtousethepackage:
Python:source:trunk/cdo/contrib/python/test/test_cdo.pyorat githu b
Ruby:source:trunk/cdo/cont rib/ruby/test/test_cdo.rborat github
Bothbindingsaretestedwiththeunixandthewin32versionofCDO.Pleasenote,th atreturningarraysbysettingreturnCdfisnottestedduetothelackofthe
correspondingnetcdflibraryonwindows.Thereareprecompiledwindowsversion ofnetcdf,butIwillnotspen ttimetogetitrunning.
Beforedoinganythingelse,thelibrariesmusthavebeenloadedintheusualway:
fromcdoimport*#pythonversion
cdo=Cdo()
Inthepythonversionanobjecthastobecreatedforinternalreasons,whereasthisisnotnecessaryforRuby.Thismaychangeinthefuture,butfornowit isonlya
minordifference
require'cdo'#rubyversion
online/offline help
Forallnonoperators,the automaticallygenerat eddocumentationformrubygemsmightbehelpful.Operatordocumentationcanbeviewedonline,directlybycalling
cdo‐h<operatorpattern>
orwithintheinteractivepython/rubyshell.Bothofthefollingexamplesdisplaythebuiltinhelpforsinfov:
Python:
fromcdoimport*
help(Cdo().sinfov)
or
fromcdoimport*
cdo=Cdo()
help(cdo.sinfov)
Ruby
require'cdo'
Cdo.help('sinfov')#orCdo.help(:sinfov)
IO
Inputandoutputfilescanbesetwiththekeyworsinputandoutput
Cdo.infov(:input=>ifile)#rubyversion
cdo.showlevel(:input=>ifile)
cdo.infov(input=ifile)#pythonverson
cdo.showlevel(input=ifile)

2019/11/4 Cdo{rbpy}CDOProjectManagementService

https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo%7Brbpy%7D 2/7

Cdo.timmin(:input=>ifile,:output=>ofile)#rubyversion

cdo.timmin(input=ifile,output=ofile)#pythonversion

Options

Commandlineoption slike'‐f'or'‐P'canbyusedviatheoptionskeyword:

Cdo.timmin(:input=>ifile,:output=>ofile,:options=>'‐fnc')#rubyversion

cdo.timmin(input=ifile,output=ofile,options='‐fnc')#pythonversion

Operatorargumentshavetobegivenasthefirstmethodargument

Cdo.remap(gridFile,weightFile,:input=>ifile,:output=>ofile,:options=>'‐fnc')#rubyversion

cdo.remap(gridFile+","+weighFile,input=ifile,output=ofile,options='‐fnc')#pythonversion

Cdo.seltimestep('1/10',:input=>ifile,:output=>ofile,:options=>'‐r‐BF64')#rubyversion

cdo.seltimestep('1/10',input=ifile,output=ofile,options='‐r‐BF64')#pythonversion

Operator Chains

TotakerealadvantageofCDOsinternalparallelism,youshouldworkwithoperatorchainsasmushaspossible:

Cdo.setname('random',:input=>"‐mul‐random,r10x10‐enlarge,r10x10‐setyear,2000‐for,1,4",:output=>ofile,:options=>'‐fnc')#rubyversion

cdo.setname('random',input="‐mul‐random,r10x10‐enlarge,r10x10‐setyear,2000‐for,1,4",output=ofile,options='‐fnc')#pythonversion

AnothergoodexampletakenfromtheTutorialillustratesthedifferentwaysofchaining:Whilethechain

cdosub‐dayavgifile2‐timavgifile1ofile

isrepresentedby

Cdo.sub(:input=>"‐dayavg#{ifile2}‐timavg#{ifile1}",:output=>ofile)#ruby

cdo.sub(input="‐dayavg"+ifile2+"‐timavg"+ifile1,output=ofile)#python

Theserialversion,whichprohibitsinternalparallelism,createsunnecessarytemporalfilesandisj ustmentionedforeducationalreasonswouldlooklike

Cdo.sub(:input=>Cdo.dayavg(:input=>ifile2)+""+Cdo.timavg(:input=>ifile1),:output=>ofile)#ruby

cdo.sub(input=cdo.dayavg(input=ifile2)+""+cdo.timavg(input=ifile1),output=ofile)#python

orusingthejoinmethod:

Cdo.sub(:input=>[Cdo.dayavg(:input=>ifile2),Cdo.timavg(:input=>ifile1)].join(""),:output=>ofile)#ruby

cdo.sub(input="".join([cdo.dayavg(input=ifile2),cdo.timavg(input=ifile1)],output=ofile)#python

Special Features

Tempfile handling

Iftheoutputstreamisomitted,atemporaryfileiswrittenanditsnameisthereturnvalueofthecall:

ofile=Cdo.timmin(:input=>ifile,:options=>'‐fnc')#rubyversion

ofile=cdo.timmin(input=ifile,options='‐fnc')#pythonversion

Here,theoutputfilesareautomaticallyremoved,whenthescriptsfinishes.Manualcleanupisnotnecessaryanymoreunlessyouencounteracrash.Inthatcaseyou

canusethenew(1.4.0)method

cdo.cleanTempDir()

orsetanalternativedirectoryforstoringtemporaryfileswith

cdo=Cdo(tempdir='/path/to/new/tempdir')#python

cdo=Cdo.new(tempdir:'/path/to/new/tempdir')#ruby

Conditional Processing

Whenprocessinglargenumberofinputfilesasitisthecaseinarunningexperiment,itcanbeveryhelpfultosuppressthecreationofintermediateoutputifthesefiles

arealreadythere.Thiscanspeedupyourpostprocessing.Inthedefaultbehavior,outputiscreatednomatterifsomethingisoverwrittenornot.Conditionalprocessing

canbeusedintwodifferentways:

globalsetting

cdo.forceOutput=False#python

Cdo.forceOutput=false#ruby