o W Jannis Leidel h
[email protected] d @enn_io
6 How I learned to stop worrying & love Python Pack Packaging aging
6 1) The past of Python packaging 2) Common pitfalls and gotchas 3) Everyday software development 4) The future of Python packaging
Ambiguous terms
- Pyth Python on pack packag age e A directory with Python files (module) and a __init__.py file - Release A version of a specific software, result of a development cycle, e.g. “Django 1.3” - Dis Distri tributi bution on Source or binary form of a release, e.g. zipped tarball or Windows installer
1 The past of Python packaging, a brief historical ov overview erview
Distutils history
“[..] a standard mechanism for building, distributing, and installing Python modules – or, more realistically, multimodule distributions.” distributions.” Greg Ward
- crea creattes dist distri ribu buti tion ons s - instal installs ls to to libr library ary dir direct ectory ory - buil builds ds C exten xtensi sion ons s - proc proces esses ses docum document entati ation on
Distutils enhancements
- PEP 241 (2001) Metadata in PKG-INFO files added to with package distributions - PEP 301 (2002) Package Index and Trove rove classifiers classifie rs added and register command - PEP 314 (2003)
Metadata 1.1 with license, platform, download download URL, dependencies fields
Distutils enhancements
- PEP 345 (2005-today)
Metadata 1.2, better dependencies - PEP 376 (2009-today)
Database of Installed Python packages - PEP 386 (2009-today)
Sane version comparison module and better version handling on PyPI
2 Common pitfalls and gotchas of packaging Python softwar software e
The standard setup.py
fromdistutils.coreimportsetup setup( name='MyApp', version='0.1.0', description='Myappthatdoesthings', author='JohnDoe', author_email='
[email protected]', license='BSD', packages=['myapp'], package_data={ 'myapp':[ 'templates/*.html', ], }, )
Long description
Rendered on PyPI with docutils reStructuredText renderer importcodecs defreadme(filename): file=codecs.open(filename,'utf-8') returnunicode(file.read()) setup( #... long_description=readme('README.rst'), )
package_data
Lists all additional, non-Python files of a release to be installed from a distribution setup( #... package_data={ 'myapp':[ 'static/myapp/*/*', 'templates/myapp/*', 'locale/*/LC 'locale/*/LC_MESSAGES/*' _MESSAGES/*' ], }, )
MANIFEST.in
Template for the “MANIFEST” file that lists all files to be put in a distribution
includeREADME.rst recursive-includedocs*.txt recursive-includemyapp/locale*.mo*.po recursive-includemyapp/static*.css*.png recursive-includemyapp/templates*.html
Use sane versions, dammit!
- 0.18 “Catty “Catty And The The Majo Major” r” - .000001 - “unr “unrel elea ease sed. d.un uno ocialdev” Don’t do that. - Use PEP 386 386 for for form formal al vers version ions, s, e.g e.g.. “1.0.1” “1.0.1” or “0.5a4” etc - Use relea release se name names s in any any prose prose documentation and changelogs, e.g. super hero names
3 Using packaging in everyday’s software development
System Syst em package management or not?
- Inst Instal alll bina binary ry pac packa kage ges s globally using your operatings system’s package management, e.g. database adapters, PIL, lxml - Vendorize packages that require modifications or short term patches - Install Install often often updated updated and your your projec projectt specific packages locally, locally, e.g. Django, pytz, your own app
virtualenv
- Isolat Isolated ed Pytho Python n envir environm onment ents s - Creat Create e by topic topic,, e.g. e.g. by by “branch “branch”, ”, “client” “client”,, “milestone”, “staging/prod” etc. - Use virtua virtualen lenvv-wr wrapper apper for even even more more shell helpers - pip included included in ev every virtual virtual envir environmen onmentt
pip
- Sane Sane instal installat lation ion and and uninsta uninstalla llatio tion n - freezi freezing ng and and unfr unfreez eezing ing of of requir requirement ements s $server1:~pipfreeze $server1:~pipfreeze>dev-deps.t >dev-deps.txt xt $server2:~pipinstall $server2:~pipinstall-rdev-deps -rdev-deps.txt .txt
- VCS integ integra ration tion for quick quick and dirty use - PyPI mirro mirrorr suppo support rt (PEP (PEP 381) 381),, e.g. e.g. d.pypi.python.org
Simple own package index
- HTTP server server with with direc directory tory index index featur feature, e, e.g. Apache’s DirectoryIndex - PyPI PyPI clon clone e Chi Chish shop op pipinstall-fhttp://localhost/dists/ pipinstall-ihttp://pypi.corp.local/
- Use Use ~/.pip ~/.pip/ /pip pip.conf onf [install] index-url=http://pypi index-url=http://pypi.corp.local/ .corp.local/
Local index
- Use Use adad-ho hoc c HTT HTTP P ser serv ver $python-mSimpleHTTPS $python-mSimpleHTTPServer8000 erver8000
- .pyd .pydis istut tutils ils..cfg cfg sdis sdistt dir dir [sdist] dist-dir=/var/www/dists
- Enable Enables s pip pip downl download oad-ca -cache che [global] download-cache=~/.pip download-cache=~/.pip-downloads -downloads
4 The future of packaging with Distutils
What is distutils2?
- toolbo toolbox x with with refer referenc ence e implemen implementatio tation n for PEP 345, PEP 376, 376, PEP 386 - the stan standal dalone one pys pysetu etup p tool tool that that installs and uninstalls distributions (e.g. “pysetup install Django”) - metada metadata ta avail availabl able e outsid outside e of Python Python file in setup.cfg - alread already y in Python Python 3 trun trunk k as “pack “packagi aging ng””
Other distutils2 features
- defined defined requ requir ireme ements nts depe dependi nding ng on environment Requires-Dist:bar;python_version== '2.4'orpython_version=='2.5' Requires-Python:>=2.5,<3
- Obso Obsole leti ting ng othe otherr rele releas ases es Obsoletes-Dist:OtherPr Obsoletes-Dist:OtherProject(<3.0) oject(<3.0)
Moving from distutils to distutils2
- “pyse “pysetup tup creat create” e” for con conv version setup setup.p .py y to setup.cfg for forward compatibility - “pyse “pysetup tup gener generat ate-setu e-setup” p” for for conv conversio ersion n setup.cfg to setup.py for backwards compatibility
Example distutils2 setup.cf setup.cfg g [metadata] name=myapp version=0.1 author=JohnDoe author-email=
[email protected] summary=Myawesomeapp description-file=README.rst home-page=http://johndoe.com/p/myapp/ project-url:Repository,http://code.johndoe.com/ myapp/ classifier=DevelopmentStatus::3-Alpha License::OSIApproved::MITLicense [files] packages=myapp extra_files=setup.pyREADME resources= etc/settings.py.dist{confdir}/myapp
Other files that are shipped with distutils2
- MET METADA ADATA - RECORD a list of installed files (CSV formatting) - RESOURCES list of non-python files - INSTALLER name of the installer - REQUESTED exists if distribution wasn’t installed as dependency
6 1) Keep your setup.py files simple 2) Follow the standards
virtualenv env/ /pip 3) Use virtual 4) Look out for distutils2/packaging 5) In doubt, read the
“Hitchhiker “Hitchhik er’s ’s Guide to Packaging”
Thanks!
Questions? Don‘t forget to
o W Jannis Leidel h
[email protected] d @enn_io
while we’re in Amsterdam!
